In my earlier article I gave an overview on systemd-journald service and how logging works with journal files in systemctl logs. Now in this journalctl cheat sheet I will show various examples to filter and view systemd logs such as Linux boot messages.
This article journalctl cheat sheet was written while using CentOS 7, so it is safe to say that it also fully covers RHEL 7, Fedora, Oracle Enterprise Linux and generally the whole Red Hat family of operating systems and possibly Novell’s SLES and OpenSUSE.
Journalctl cheat sheet
Logging data is collected, stored, and processed by the Journal's systemd-journald service. It creates and maintains binary files called journals based on logging information that is received from the kernel, from user processes, from standard output, and standard error output of system services or via its native API. These journals are structured and indexed, which provides relatively fast seek times.
To view all of the systemd-journald collected data, the journalctl utility is used. This command provides several means of filtering the data, and it is an extremely powerful utility. In this article journalctl cheat sheet we will try to explore all the areas to filter systemd logs
1. Viewing systemctl log files without any arguments
Let is start with our journalctl cheat sheet, To access the systemd-journald logs, use the journalctl tool without any argument. For a basic view of the logs type as root:
# journalctl
An output of this command is a list of all log files generated on the system including messages generated by system components and by users. The structure of this output is similar to one used in /var/log/messages/ but with certain improvements:
2. View journal logs runtime
You can view systemd logs runtime similar to tail -f using journalctl -f. This opens the live view mode of systemd-journald, which allows you to see new messages scrolling by in real time. Use Ctrl+C to interrupt.
# journalctl -f
3. List all the Linux boot messages using numerical identifiers
To view Linux boot messages pertaining to previous boots; this can be viewed by passing the --list-boots option:
# journalctl --list-boots 0 60f1ed8ef60c49fa80e9fed06ab93dec Sun 2019-09-01 07:35:02 IST—Fri 2019-09-06 10:23:36 IST
Based on the preceding output for Linux boot messages, we can see seven files that contain Linux boot messages; we can view any of these files by passing the offset of the file. The offset of each file is the value in the first column. Since we have only one file, let's take a look at Linux boot messages from the 0 offset:
# journalctl -b 0
-- Logs begin at Sun 2019-09-01 07:35:02 IST, end at Fri 2019-09-06 10:23:55 IST. --
Sep 01 07:35:02 rhel-7.example CROND[28806]: (root) CMD (/opt/cntdb/bin/check_ams_ntpd_services)
Sep 01 07:35:02 rhel-7.example rsyslogd[27197]: imjournal: journal reloaded... [v8.24.0-34.el7 try http://www.rsyslog.com/e/0 ]
Sep 01 07:35:02 rhel-7.example systemd[1]: Removed slice User Slice of root.
Sep 01 07:35:04 rhel-7.example su[28854]: (to dbmrun) root on none
Sep 01 07:35:04 rhel-7.example systemd[1]: Started Session c1558 of user dbmrun.
Here the Linux boot messages begin at Sun 2019-09-01
4. Filter systemd logs based on timestamp
You can view systemd logs based on timestamp. There are various arguments to filter such messages. Some of them are shown below:
# journalctl --since today
-- Logs begin at Thu 2019-09-05 15:04:51 IST, end at Fri 2019-09-06 10:25:39 IST. --
Sep 06 00:00:01 rhel-7.example systemd[1]: Created slice User Slice of root.
Sep 06 00:00:01 rhel-7.example systemd[1]: Started Session 1457 of user root.
Sep 06 00:00:01 rhel-7.example systemd[1]: Started Session 1459 of user root.
Sep 06 00:00:01 rhel-7.example systemd[1]: Created slice User Slice of provgw.
# date Fri Sep 6 10:25:57 IST 2019
Check systemd-journald logs for certain day and time
# journalctl --since "2019-08-26 15:00:00"
-- Logs begin at Thu 2019-08-22 15:08:47 IST, end at Fri 2019-09-06 14:07:28 IST. --
Aug 26 15:00:01 rhel-7.example systemd[1]: Started Session 1844 of user root.
Aug 26 15:00:01 rhel-7.example systemd[1]: Starting Session 1844 of user root.
Aug 26 15:00:01 rhel-7.example systemd[1]: Started Session 1845 of user root.
Aug 26 15:00:01 rhel-7.example systemd[1]: Starting Session 1845 of user root.
Check systemd-journald logs between a time period
# journalctl --since "2019-08-26 15:00:00" --until "2019-08-27 15:00:00" -- Logs begin at Thu 2019-08-22 15:08:47 IST, end at Fri 2019-09-06 14:15:03 IST. -- Aug 26 15:00:01 rhel-7.example systemd[1]: Started Session 1844 of user root. Aug 26 15:00:01 rhel-7.example systemd[1]: Starting Session 1844 of user root. <Output trimmed> Aug 27 14:56:02 rhel-7.example su[26926]: pam_unix(su-l:session): session opened for user root by oamsys(uid=1000) Aug 27 14:56:02 rhel-7.example unknown[26949]: bash root 26927: User root logged from 10.43.143.55 Aug 27 14:56:02 rhel-7.example unknown[27077]: bash root 26927: User root logged from 10.43.143.55 Aug 27 14:56:02 rhel-7.example unknown[27125]: bash root 26927: User root logged from 10.43.143.55
Also to check systemd-journald logs between yesterday and today
# journalctl --since yesterday --until now
-- Logs begin at Thu 2019-09-05 15:07:24 IST, end at Fri 2019-09-06 14:51:20 IST. --
Sep 05 15:07:24 rhel-7.example GoExample[4148]: 2019/09/05 15:07:24 Test message 684
Sep 05 15:07:24 rhel-7.example GoExample[4148]: 2019/09/05 15:07:24 Test message 685
Sep 05 15:07:24 rhel-7.example GoExample[4148]: 2019/09/05 15:07:24 Test message 686
5. Filter messages based on unit file (for eg: systemd-journald)
We can pass the -u option and specify which service we're looking for:
Here we are checking all the logs for systemd-journald service
# journalctl -u systemd-journald
-- Logs begin at Thu 2019-08-22 15:08:47 IST, end at Fri 2019-09-06 14:08:30 IST. --
Aug 22 15:08:47 rhel-7.example systemd-journal[267]: Runtime journal is using 8.0M (max allowed 4.0G, trying to leave 4.0G free
Aug 22 15:08:47 rhel-7.example systemd-journal[267]: Journal started
Aug 22 15:08:52 rhel-7.example systemd-journal[267]: Journal stopped
To check SSHD service logs
# journalctl -u sshd.service
-- Logs begin at Thu 2019-08-22 15:08:47 IST, end at Fri 2019-09-06 14:09:19 IST. --
Aug 22 15:09:05 rhel-7.example systemd[1]: Starting OpenSSH server daemon...
Aug 22 15:09:05 rhel-7.example sshd[2034]: Server listening on 10.43.138.1 port 22.
Aug 22 15:09:05 rhel-7.example systemd[1]: Started OpenSSH server daemon.
Aug 22 15:22:33 rhel-7.example sshd[6734]: Connection from 10.43.143.55 port 53350 on 10.43.138.1 port 22
Alternatively you can also use _SYSTEMD_UNIT
# journalctl _SYSTEMD_UNIT=sshd.service
-- Logs begin at Thu 2019-08-22 15:08:47 IST, end at Fri 2019-09-06 14:23:30 IST. --
Aug 22 15:09:05 rhel-7.example sshd[2034]: Server listening on 10.43.138.1 port 22.
Aug 22 15:22:33 rhel-7.example sshd[6734]: Connection from 10.43.143.55 port 53350 on 10.43.138.1 port 22
6. Filter logs based on binary file
What we also get in the preceding example are the systemd logs that interact with the sshd unit. But if we only want logs from the sshd binary
# journalctl /sbin/sshd
-- Logs begin at Thu 2019-08-22 15:08:47 IST, end at Fri 2019-09-06 14:17:04 IST. --
Aug 22 15:09:05 rhel-7.example sshd[2034]: Server listening on 10.43.138.1 port 22.
Aug 22 15:22:33 rhel-7.example sshd[6734]: Connection from 10.43.143.55 port 53350 on 10.43.138.1 port 22
Aug 22 15:22:33 rhel-7.example sshd[6734]: Postponed keyboard-interactive for oamsys from 10.43.143.55 port 53350 ssh2 [preauth]
7. Filter logs with more details
Let us continue with our journalctl cheat sheet. We can also use -x with journalctl command to augment log lines with explanation texts from the message catalog. This will add explanatory help texts to log messages in the output where this is available.
# journalctl -u sshd.service -x
-- Logs begin at Thu 2019-08-22 15:08:47 IST, end at Fri 2019-09-06 14:18:31 IST. --
Aug 22 15:09:05 rhel-7.example systemd[1]: Starting OpenSSH server daemon...
-- Subject: Unit sshd.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit sshd.service has begun starting up.
Aug 22 15:09:05 rhel-7.example sshd[2034]: Server listening on 10.43.138.1 port 22.
Aug 22 15:09:05 rhel-7.example systemd[1]: Started OpenSSH server daemon.
-- Subject: Unit sshd.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit sshd.service has finished starting up.
--
-- The start-up result is done.
Aug 22 15:22:33 rhel-7.example sshd[6734]: Connection from 10.43.143.55 port 53350 on 10.43.138.1 port 22
8. Filter logs based on Process PID
To view systemd logs based on process PID we can use _PID as shown below:
# journalctl _PID=26730
-- Logs begin at Thu 2019-08-22 15:08:47 IST, end at Fri 2019-09-06 14:21:43 IST. --
Aug 27 14:56:01 rhel-7.example sshd[26730]: Connection from 10.43.143.55 port 49391 on 10.43.138.1 port 22
Aug 27 14:56:01 rhel-7.example sshd[26730]: Postponed keyboard-interactive for oamsys from 10.43.143.55 port 49391 ssh2 [preauth]
Aug 27 14:56:01 rhel-7.example sshd[26730]: Postponed keyboard-interactive/pam for oamsys from 10.43.143.55 port 49391 ssh2 [prea
Aug 27 14:56:01 rhel-7.example sshd[26730]: Accepted keyboard-interactive/pam for oamsys from 10.43.143.55 port 49391 ssh2
Aug 27 14:56:01 rhel-7.example sshd[26730]: pam_unix(sshd:session): session opened for user oamsys by (uid=0)
Aug 27 14:56:01 rhel-7.example sshd[26730]: User child is on pid 26733
9. Filter logs based on priority
Filter output by message priorities or priority ranges. Takes either a single numeric or textual log level (i.e. between 0/"emerg" and 7/"debug"), or a range of numeric/text log levels in the form FROM..TO. The log levels are the usual syslog log levels i.e. "emerg" (0), "alert" (1), "crit" (2), "err" (3), "warning" (4), "notice" (5), "info" (6), "debug" (7).
Here I have filtered logs for "emerg" priority.
# journalctl -p 0
-- Logs begin at Thu 2019-08-22 15:08:47 IST, end at Fri 2019-09-06 14:28:07 IST. --
Sep 03 20:00:02 rhel-7.example dracut[15798]: Will not override existing initramfs (/tmp/test/initrd.img) without --force
Sep 03 20:04:47 rhel-7.example dracut[30438]: Will not override existing initramfs (/tmp/test/initrd.img) without --force
Sep 03 20:15:27 rhel-7.example dracut[17519]: Will not override existing initramfs (/tmp/test/initrd.img) without --force
Here we are filtering logs for a range between emerg(0) and critical(2)
# journalctl -p 0..2
-- Logs begin at Thu 2019-08-22 15:08:47 IST, end at Fri 2019-09-06 14:28:23 IST. --
Aug 23 03:29:11 rhel-7.example binary-check[11670]: ALERT Load check failed, the load might have been compromised! Detail follows
Aug 23 03:29:11 rhel-7.example binary-check[11671]: ALERT /usr/bin/dbus-cleanup-sockets: FAILED. This file has been modified!
Aug 23 03:29:11 rhel-7.example binary-check[11672]: ALERT /usr/bin/dbus-daemon: FAILED. This file has been modified!
10. Filter kernel messages
In this journalctl cheat sheet to view only kernel generated systemd logs, you can use (-k). It is equivalent to (--dmesg). To some extent you can also view Linux boot messages from kernel.
# journalctl -k
-- Logs begin at Thu 2019-08-22 15:08:47 IST, end at Fri 2019-09-06 14:31:08 IST. --
Aug 22 15:08:47 rhel-7.example kernel: Initializing cgroup subsys cpuset
Aug 22 15:08:47 rhel-7.example kernel: Initializing cgroup subsys cpu
Aug 22 15:08:47 rhel-7.example kernel: Initializing cgroup subsys cpuacct
Or alternatively you can also use _TRANSPORT where all the logs with "kernel" match will be filtered
# journalctl _TRANSPORT=kernel
-- Logs begin at Thu 2019-08-22 15:08:47 IST, end at Fri 2019-09-06 14:31:53 IST. --
Aug 22 15:08:47 rhel-7.example kernel: Initializing cgroup subsys cpuset
Aug 22 15:08:47 rhel-7.example kernel: Initializing cgroup subsys cpu
Aug 22 15:08:47 rhel-7.example kernel: Initializing cgroup subsys cpuacct
11. Check the disk or memory used by journal logs
If you are using persistent storage then the below output shows the amount of disk used and if using non-persistent storage then this command will show the amount of memory used for systemd logs.
# journalctl --disk-usage
Archived and active journals take up 384.0M on disk.
12. Perform journal log files cleanup
You can use --vaccum-size which removes archived journal files until the disk space they use falls below the specified size (specified with the usual "K", "M", "G", "T" suffixes),
# journalctl --disk-usage
Archived and active journals take up 384.0M on disk.
We will reduce our journal file usage to 200MB using below command:
# journalctl --vacuum-size=200M
Deleted archived journal /run/log/journal/9bf0fc6f60ce41149a0a540f52db9773/system@822262d9c350464d9b67a3fd4fa4b8d8-00000000003bf4d9-000591cb0a1ae4c4.journal (40.0M).
Deleted archived journal /run/log/journal/9bf0fc6f60ce41149a0a540f52db9773/system@822262d9c350464d9b67a3fd4fa4b8d8-00000000003c7a18-000591cb0bb46d64.journal (40.0M).
Deleted archived journal /run/log/journal/9bf0fc6f60ce41149a0a540f52db9773/system@822262d9c350464d9b67a3fd4fa4b8d8-00000000003cff37-000591cb0d3dbe9b.journal (40.0M).
Deleted archived journal /run/log/journal/9bf0fc6f60ce41149a0a540f52db9773/system@822262d9c350464d9b67a3fd4fa4b8d8-00000000003d847a-000591cb0ec7bb1a.journal (40.0M).
Deleted archived journal /run/log/journal/9bf0fc6f60ce41149a0a540f52db9773/system@822262d9c350464d9b67a3fd4fa4b8d8-00000000003e0998-000591cb1060df61.journal (24.0M).
Vacuuming done, freed 184.0M of archived journals on disk.
Now check the disk/memory usage journal logs
# journalctl --disk-usage
Archived and active journals take up 200.0M on disk.
Alternatively you can also use --vacuum-time or you can use both --vaccum-size and --vacuum-time together to enforce both a size and time limit on the archived journal files
--vaccum-time is specified with the usual "s", "min", "h", "days", "months", "weeks", "years" suffixes
13. Filter logs based on user
To view systemd-journald logs for individual user, you can use USERID to view the journal data concerning the mapped user. For example I have a user 'deepak' with user id 1008
# id oamsys uid=1008(deepak) gid=100(users) groups=100(users),10(wheel)
So to filter the systemd logs for this user I can use below command:
# journalctl _UID=1008
-- Logs begin at Thu 2019-09-05 15:07:24 IST, end at Fri 2019-09-06 14:43:49 IST. --
Sep 06 11:26:21 rhel-7.example sshd[20525]: Starting session: shell on pts/3 for oamsys from 10.136.206.251 port 60337 id 0
Sep 06 11:26:23 rhel-7.example USERACT[20723]: pts/3, oamsys, su
Sep 06 11:26:25 rhel-7.example su[20724]: (to root) oamsys on pts/3
Sep 06 11:26:25 rhel-7.example su[20724]: pam_unix(su:session): session opened for user root by oamsys(uid=1008)
Sep 06 12:16:19 rhel-7.example sshd[20525]: Read error from remote host 10.136.206.251 port 60337: Connection timed out
Sep 06 12:16:19 rhel-7.example su[20724]: pam_unix(su:session): session closed for user root
14. View logs using journalctl in verbose mode
To view systemd logs in verbose mode use -o verbose
# journalctl -o verbose
-- Logs begin at Mon 2019-09-02 16:16:39 IST, end at Wed 2019-09-04 04:22:31 IST. --
Mon 2019-09-02 16:16:39.603861 IST [s=ce2e96e21a954a1fa83a102c90207756;i=1;b=7b6f05a323bf4d8fb43149c8fc7c95c0;m=186bd5;t=5918f
PRIORITY=6
_TRANSPORT=driver
MESSAGE=Runtime journal is using 8.0M (max allowed 189.4M, trying to leave 284.2M free of 1.8G available → current limit 1
MESSAGE_ID=ec387f577b844b8fa948f33cad9a75e6
_PID=87
_UID=0
_GID=0
_COMM=systemd-journal
_EXE=/usr/lib/systemd/systemd-journald
_CMDLINE=/usr/lib/systemd/systemd-journald
_CAP_EFFECTIVE=5402800cf
_SYSTEMD_CGROUP=/system.slice/systemd-journald.service
_SYSTEMD_UNIT=systemd-journald.service
_SYSTEMD_SLICE=system.slice
_BOOT_ID=7b6f05a323bf4d8fb43149c8fc7c95c0
_MACHINE_ID=3a0d751560f045428773cbf4c1769a5c
_HOSTNAME=master.example.com
Mon 2019-09-02 16:16:39.603953 IST [s=ce2e96e21a954a1fa83a102c90207756;i=2;b=7b6f05a323bf4d8fb43149c8fc7c95c0;m=186c32;t=5918f
PRIORITY=6
_BOOT_ID=7b6f05a323bf4d8fb43149c8fc7c95c0
_MACHINE_ID=3a0d751560f045428773cbf4c1769a5c
I hope the journalctl cheat sheet was helpful. I have shown examples for individual arguments to filter logs but you can also combine them to improve visibility and filtering to a greater extent. There are many more arguments which can be used to view systemd logs which you can collect from the man page of journalctl.
very useful information! thanks