Troubleshooting systemd Services with systemctl and journalctl

When a Linux service fails under systemd, the fastest path to a fix is usually not restarting it repeatedly. A better approach is to inspect the service state, read the relevant logs, confirm the effective unit configuration, and validate changes before trying again.

This guide presents a repeatable troubleshooting workflow for modern Linux distributions that use systemd. The examples use commands available in systemd 257, but the core workflow also applies to many earlier systemd releases.

Start with the Service Status

Begin with systemctl status:

sudo systemctl status myapp.service --no-pager

The status output gives you several useful signals in one place:

• whether the unit is active, inactive, or failed;
• the main process ID when the service is running;
• the exit code or signal when it failed;
• the unit file path;
• a small set of recent log messages.

Pay particular attention to the Active: and Process: lines. For example, an exit status such as status=203/EXEC usually points to a problem starting the configured executable, while a normal application exit code such as status=1/FAILURE means the process started but returned an error.

Do not treat the short log excerpt from systemctl status as the complete diagnostic record. It is only a starting point.

Read the Full Logs with journalctl

To inspect messages for one service from the current boot, use:

sudo journalctl -u myapp.service -b --no-pager

The -u option filters by unit, while -b limits results to the current boot. This avoids mixing current failures with older incidents.

If the failure happened recently, narrow the time range:

sudo journalctl -u myapp.service --since "10 minutes ago" --no-pager

Useful error patterns include:

• No such file or directory for incorrect executable or file paths;
• Permission denied for filesystem or execution permission problems;
• missing environment variables or configuration files;
• connection failures to databases, sockets, or upstream services;
• application stack traces or explicit fatal errors.

A practical habit is to look for the first meaningful error rather than the last line. Later messages are often consequences of the original failure.

Inspect the Effective Unit Configuration

A service may not be using the unit file you expect. Package updates, local overrides, and drop-in files can change the effective configuration.

Inspect the complete unit as systemd sees it:

sudo systemctl cat myapp.service

This command shows the main unit file and any drop-in configuration files. Check fields such as:

[Service]
User=myapp
Group=myapp
WorkingDirectory=/srv/myapp
ExecStart=/srv/myapp/bin/server
EnvironmentFile=/etc/myapp/myapp.env

Common mistakes include an invalid WorkingDirectory, a misspelled EnvironmentFile, or an ExecStart path that existed during deployment but no longer exists.

You can also query selected effective properties directly:

sudo systemctl show myapp.service -p ExecStart -p EnvironmentFiles -p User -p Group -p WorkingDirectory

This is useful when a unit contains several overrides and you only need to confirm the final values systemd is using.

Verify Paths and Permissions from the Service User’s Perspective

A command that works in your interactive shell can still fail as a service because systemd may run it as another user with a different working directory and environment.

Check the configured executable and working directory:

ls -l /srv/myapp/bin/server
ls -ld /srv/myapp

If the service runs as myapp, test whether that user can access the executable:

sudo -u myapp test -x /srv/myapp/bin/server

A successful test command produces no output and exits with status 0. If it fails, inspect permissions on every directory in the path, not only the executable itself.

For example, a binary can be executable while still being unreachable because one parent directory does not grant the service user traversal permission.

Check Environment Files Carefully

Environment-related failures are common because an interactive shell may define variables that do not exist inside a service.

If your unit contains:

[Service]
EnvironmentFile=/etc/myapp/myapp.env

confirm that the file exists and is readable by the service setup:

sudo test -r /etc/myapp/myapp.env

Keep secrets out of commands, logs, screenshots, and troubleshooting notes. When checking environment configuration, verify variable names and file permissions without printing sensitive values unnecessarily.

If you only need non-sensitive settings, a drop-in can define them explicitly:

[Service]
Environment="APP_ENV=production"
Environment="APP_PORT=8080"

Create overrides with systemctl edit rather than modifying a vendor-provided unit file in /usr/lib/systemd/system or /lib/systemd/system directly.

Validate Unit Syntax Before Restarting

Before reloading systemd, validate a unit file with systemd-analyze verify:

sudo systemd-analyze verify /etc/systemd/system/myapp.service

This can catch invalid directives, malformed values, and some dependency or executable problems before you restart the service.

A minimal valid service for troubleshooting experiments might look like this:

[Unit]
Description=Nalar systemd troubleshooting demo
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/bash -c 'echo "demo started"; exit 1'
Restart=no

[Install]
WantedBy=multi-user.target

The unit is intentionally configured to exit with status 1. That makes it useful for practicing the inspection workflow without depending on an application, database, or network service.

After editing a real unit or drop-in, reload the manager configuration:

sudo systemctl daemon-reload

Then restart the service:

sudo systemctl restart myapp.service

Immediately check both status and logs again. A configuration change is not complete until the observed runtime behavior matches the intended result.

Understand Restart Loops

A service with an aggressive restart policy can fail repeatedly and produce a large amount of log noise.

For example:

[Service]
Restart=on-failure
RestartSec=2s

This is useful for transient failures, but it can hide the original error behind many repeated attempts.

When investigating a restart loop, inspect the earliest failure in the relevant time window. If systemd eventually reports start request repeated too quickly, fix the underlying error first, then clear the failed state if necessary:

sudo systemctl reset-failed myapp.service

Resetting the failed state does not fix the service. It only clears systemd’s recorded failure counters and state.

Check Dependencies When the Service Itself Looks Correct

Sometimes the application configuration is valid, but a dependency is unavailable.

Inspect declared dependencies with:

sudo systemctl list-dependencies myapp.service

Then check the relevant dependency directly. For example, if the application needs a local database service:

sudo systemctl status postgresql.service --no-pager

Be careful with assumptions about ordering. After=network.target controls startup ordering; it does not guarantee that a remote API, DNS server, or database endpoint is ready to accept connections.

Applications should still handle transient dependency failures sensibly, even when systemd unit ordering is configured correctly.

A Repeatable Troubleshooting Sequence

When a service fails, use this order:

1. Run systemctl status to identify the high-level failure state.
2. Read unit-specific logs with journalctl -u.
3. Inspect the effective configuration with systemctl cat and systemctl show.
4. Verify executable paths, working directories, users, permissions, and environment files.
5. Validate edited unit files with systemd-analyze verify.
6. Run systemctl daemon-reload after unit changes.
7. Restart the service and confirm the new behavior in both status and logs.
8. Check dependencies if the service configuration is correct but startup still fails.

This sequence works because it moves from evidence to configuration to controlled changes. It reduces guesswork and makes it easier to identify the original failure instead of reacting to secondary symptoms.

Conclusion

Most systemd failures become much easier to diagnose when you combine systemctl status, targeted journalctl queries, effective unit inspection, permission checks, and systemd-analyze verify.

The important principle is to troubleshoot from evidence. Read the first useful error, confirm what systemd is actually executing, change one relevant thing at a time, and verify the result after every restart. That workflow is faster and safer than repeatedly restarting a broken service without understanding why it failed.