journalctl
journalctl is the command-line tool that reads the systemd journal, the structured binary log kept by systemd-journald. It filters entries by service unit, priority, boot and time range, follows new messages live, and prints them as text or JSON. On most modern Linux servers it is the first place to look when a service fails or someone tries to log in.
What journalctl reads and how the journal is stored
systemd-journald collects messages from the kernel, from services and from anything that logs through syslog or standard output, and stores them as indexed binary records. journalctl is the viewer for that store. Each record is a set of fields. Fields such as MESSAGE and PRIORITY are supplied by the program that logged. Fields that start with an underscore, such as _SYSTEMD_UNIT, _PID, _UID and _EXE, are added by journald itself and client code cannot alter them, so they are more reliable in an investigation than the message text (systemd.journal-fields(7)).
Where the journal lives is set by Storage= in journald.conf(5). volatile keeps it in memory under /run/log/journal, persistent writes it to /var/log/journal, and auto behaves like persistent only if that directory already exists. The built-in default is chosen when systemd is compiled, so do not rely on it: set the value explicitly. Reading the system journal requires root or membership in the systemd-journal, adm or wheel group, as described in journalctl(1).
Filtering the journal: the commands you will actually use
Most work is a combination of a few filters. -u selects a unit, -p a priority (one level includes everything more severe, and FROM..TO gives a range), -S and -U a time window, -b a boot (-b -1 is the previous one), -k kernel messages and -g a pattern matched against the message. -f prints the latest entries and keeps following new ones.
journalctl -u ssh --since "1 hour ago" -p warning
journalctl -u ssh --since yesterday --until today -g "Failed|Invalid user"
journalctl -b -1 -k
journalctl -f -u nginx
journalctl -u ssh -n 5 -o json-pretty --no-pagerThe SSH unit is named ssh on Debian and Ubuntu and sshd on the RHEL family. Output formats chosen with -o include short (the default), short-iso, verbose, json, json-pretty, export and cat. -F _SYSTEMD_UNIT lists every unit that has logged, which is a quick way to see what is talking. Add --utc when you compare timestamps with logs from other servers.
Retention, rate limits and what the journal cannot prove
A volatile journal vanishes at reboot, which is exactly when you need it after a compromise. Make storage persistent and bound its size with a drop-in file, then restart systemd-journald:
# /etc/systemd/journald.conf.d/50-persistent.conf
[Journal]
Storage=persistent
SystemMaxUse=1G
MaxRetentionSec=3monthWithout SystemMaxUse= the journal may use up to 10% of its filesystem and keeps 15% free, each value capped at 4G. Check usage with journalctl --disk-usage and trim archived files with --vacuum-size=, --vacuum-time= or --vacuum-files=. Active files are never removed, and --rotate archives them first. journald also rate-limits each service, by default 10000 messages per 30 seconds, so a very noisy event can push out its own evidence.
Know the limits of trust. Root can delete the journal, and Forward Secure Sealing helps you detect later modification only if you created a key with --setup-keys; journalctl --verify then checks the files. A local journal is evidence, not proof.
Using the journal for monitoring and incident response
On a systemd server the journal is the main local source for authentication and service events, so a monitoring routine starts there: a daily look at journalctl -u ssh -p warning --since yesterday, and attention to any new Accepted login from an unfamiliar address. Our monitoring guide lists what to watch, and the SSH brute force guide explains what failed-login noise means and how fail2ban blocks it.
Because whoever gets root can wipe a local journal, keep a copy elsewhere. Pass messages to rsyslog for forwarding, or follow the small server SIEM guide. When you suspect a breach, export the journal before you change anything (journalctl -o export with the output redirected to a file), copy it off the server, and then follow the breach response guide. For what the journal does not record, such as which user ran which command or touched which file, add auditd.
Related terms: rsyslog, auditd, SIEM, fail2ban, Incident response.
Frequently asked questions
What does journalctl do?
journalctl reads the systemd journal and prints its entries. You can filter by service, priority, boot or time range, follow new messages live, and export them as text or JSON.
How do I see the logs of one service with journalctl?
Run journalctl -u name, for example journalctl -u nginx, and add --since, -p or -f to narrow the time, severity or follow live output. The unit name can be a pattern, and the flag can be repeated.
Why are my journalctl logs gone after a reboot?
The journal is probably volatile and stored in memory under /run/log/journal. Set Storage=persistent in journald.conf (or a drop-in file) so it is written to /var/log/journal.
How do I reduce the disk space used by the journal?
Check usage with journalctl --disk-usage, then remove archived files with --vacuum-size=, --vacuum-time= or --vacuum-files=. For a permanent limit set SystemMaxUse= in journald.conf.
How can I check failed SSH logins with journalctl?
Query the SSH unit and search the message text, for example journalctl -u ssh --since yesterday -g "Failed|Invalid user". Use the unit name sshd on the RHEL family.
Want to see where your own server stands?
Run the free, read-only server check, or open the Security Lab and watch the detect, contain, recover, verify loop in action.
Get your free server checkOpen the Security Lab