Understanding systemd-journald and how logging works with Journal in RHEL 7

In this article I am going to give you an overview on systemd-journald service, systemd-journald logrotate functionality of Journal Logfiles, compare briefly systemd-journald vs rsyslod. Below topics will be covered in different parts of this article

• How logging works with Journal files and systemd-journald logrotate
• What is the difference between storage type as auto and persistent for journald
• How to perform log rotate of journal logs
• Systemd-journald vs rsyslogd
• Difference between RuntimeMaxUse and SystemMaxUse in journald.conf
• Which value is given preference between RuntimeMaxUse and RuntimeKeepFree, or SystemMaxUse and SystemKeepFree in journald.conf

Types of Logging Mechanism

Log files are files that contain messages about the system, including the kernel, services, and applications running on it. There are different log files for different information. For example, there is a default system log file, a log file just for security messages, and a log file for cron tasks.

Now there are three different approaches which can be used by services to write log information:

• Direct write: Some services write logging information directly to the log files, even some important services such as the Apache web server and the Samba file server.
• rsyslogd: rsyslogd is the enhancement of syslogd, a service that takes care of managing centralized log files. Syslogd has been around for a long time.
• journald: With the introduction of systemd, the journald log service systemd-journald has been introduced also. This service is tightly integrated with systemd, which allows administrators to read detailed information from the journal while monitoring service status using the systemctl status command.

Traditional syslogd daemon

Most Linux systems employ a special daemon to handle log maintenance in a unified way. The traditional Linux daemon log manager is syslogd, which is often installed from a package called sysklogd. The syslogd daemon handles messages from servers and other user-mode programs. It's usually paired with a daemon called klogd, which is generally installed from the same software package as syslogd. The klogd daemon manages logging of kernel messages.

Other choices for system loggers exist. For instance, syslog-ng is a replacement that supports advanced options. Many distributions use rsyslogd, which provides improved filtering capabilities and higher performance.

systemd-journald vs rsyslogd

• Journald provides structure and indexed log files (called journals) in a secure manner. Therefore, not only are the journal files easier to search, it is harder for system intruders to cover their tracks.. Also you can maintain rsyslog messages in structured format.
• Another advantage of using the systemd-journald service over traditional logging daemons is that journal files are automatically rotated if they grow above certain limits. This reduces log file maintenance issues and complexity.
• The journalctl utility is used to read the journal binary log files and this command provides several means of filtering the data which is very powerful.

By default, these two logging tools coexist on your RHEL 7 system.

• The journald daemon is the primary tool for troubleshooting. It also provides additional data necessary for creating structured log messages.
• Data acquired by journald is forwarded into the /run/systemd/journal/syslog socket that may be used by rsyslogd to process the data further.
• However, rsyslog does the actual integration by default via the imjournal input module, thus avoiding the aforementioned socket.

# grep imjournal /etc/rsyslog.conf
$ModLoad imjournal # provides access to the systemd journal

# local messages are retrieved through imjournal now.
$IMJournalStateFile imjournal.state

Understanding systemd journald

The systemd-journald daemon is a system service that brings together and stores logging data. Journal entries may come from several sources. The journal entries are created from server messages, user-mode program messages, and kernel messages just like the messages the syslogd daemon collects. In addition, however, journal entries are created from all system service messages, such as generated error messages and boot time communications. The systemd journal service can store journal entries regardless of their size, metadata, or format.

This event journal is stored in a binary format, and it can be queried using the journalctl command.

You can check these binary journal files under /run/log/journal or /var/log/journal based on your journald storage type as shown below

# ls -l /run/log/journal/9bf0fc6f60ce41149a0a540f52db9773/
total 57600
-rw-r-----+ 1 root systemd-journal 6553600 Sep  5 11:57 system@e6620ce9c60b4b9387ed5e5bb507638a-000000000014f164-000591c86adda67c.journal
-rw-r-----+ 1 root systemd-journal 6553600 Sep  5 11:57 system@e6620ce9c60b4b9387ed5e5bb507638a-00000000001501fe-000591c86b188161.journal
-rw-r-----+ 1 root systemd-journal 6553600 Sep  5 11:57 system@e6620ce9c60b4b9387ed5e5bb507638a-0000000000151299-000591c86b43bdd0.journal
-rw-r-----+ 1 root systemd-journal 6553600 Sep  5 11:57 system@e6620ce9c60b4b9387ed5e5bb507638a-0000000000152332-000591c86b7e948f.journal
-rw-r-----+ 1 root systemd-journal 6553600 Sep  5 11:57 system@e6620ce9c60b4b9387ed5e5bb507638a-00000000001533cd-000591c86ba9d162.journal
-rw-r-----+ 1 root systemd-journal 6553600 Sep  5 11:57 system@e6620ce9c60b4b9387ed5e5bb507638a-0000000000154466-000591c86bd50274.journal
-rw-r-----+ 1 root systemd-journal 6553600 Sep  5 11:57 system@e6620ce9c60b4b9387ed5e5bb507638a-0000000000155500-000591c86c0fdb78.journal
-rw-r-----+ 1 root systemd-journal 6553600 Sep  5 11:58 system@e6620ce9c60b4b9387ed5e5bb507638a-0000000000156591-000591c86c3add8d.journal
-rw-r-----+ 1 root systemd-journal 6553600 Sep  5 12:10 system.journal

Exploring systemd-journald system

The three primary systemd journal service components are described here:

• Daemon: The systemd journaling service is handled by the systemd-journald daemon.
• Configuration file: The journal service's configuration is set in the /etc/systemd/journald.conf file.
• Search utility: The program used to search through the journal log files is journalctl.

What will happen if I delete system.journal file?

Assuming systemd-journald service is running, the system.journal file would be the current file where journald will be writing logs directly at any time. If by any chance someone accidentally removes system.journal then you will loose all the currently "being written" logs and as soon as new log generated, system.journal file will be created automatically to write the logs.

If systemd-journald service is not in running state, then you will loose the logs. Even if system.journal exists but if service itself is not running then there would be no logs recorded by journald and no logs will be collected by rsyslog as well.

This file will be re-generated when you restart the service.

Different types of Storage supported with journald.conf

• The storage type value is controlled using /etc/systemd/journald.conf file. Under [Journal] you can modify the storage type.
• The supported values are "volatile", "persistent", "auto" and "none"
• Default storage type is configured as "auto"
• If "volatile", journal log data will be stored only in memory, i.e. below the /run/log/journal hierarchy (which is created if needed)
• If "persistent", data will be stored preferably on disk, i.e. below the /var/log/journal hierarchy, with a fallback to /run/log/journal during early boot stage and if the disk is not writable
• The "auto" value will configure journald to store journal log data in the /var/log/journal/ directory. However, the directory must already exist and have the proper permissions set. If it does not exist, then journal data is stored in the volatile /run/log/journal/ directory, and the data is erased when the system shuts down.
• "none" turns off all storage, all log data received will be dropped.

Perform systemd-journald logrotate for Journal Files

systemd-journald logrotate for journal log files will be performed based on below values

SystemMaxUse=
SystemKeepFree=
RuntimeMaxUse=
RuntimeKeepFree=
MaxFileSec=

Here,

• RuntimeMaxUse/SystemMaxUse= control how much disk space the journal may use up at maximum after which systemd-journald logrotate will be performed for journal files. This defaults to 10% of the total Physical Memory allotted to the node
• RuntimeKeepFree/SystemKeepFree= control how much disk space systemd-journald shall leave free for other uses after which systemd-journald logrotate will be performed for journal files. This default to 15% of the total Physical Memory allotted to the node

systemd-journald logrotate will respect both limits and use the smaller of the two values.

• If the file system is nearly full and either SystemKeepFree= or RuntimeKeepFree= are violated when systemd-journald is started, the limit will be raised to the percentage that is actually free.
• This means that if there was enough free space before and journal files were created, and subsequently something else causes the file system to fill up, journald will stop using more space, but it will not be removing existing files to reduce the footprint again, either i.e. no systemd-journald logrotate will be performed
• Also note that only archived files are deleted by system-journald logrotate to reduce the space occupied by journal files. This means that, in effect, there might still be more space used than SystemMaxUse= or RuntimeMaxUse= limit after a vacuuming operation is complete by systemd-journald logrotate for journal log files.

If we check the status of systemd-journald, then we can see it reports that the logs have been rotated

# systemctl status systemd-journald
● systemd-journald.service - Journal Service
   Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static; vendor preset: disabled)
   Active: active (running) since Thu 2019-09-05 15:03:38 IST; 19h ago
     Docs: man:systemd-journald.service(8)
           man:journald.conf(5)
 Main PID: 29141 (systemd-journal)
   Status: "Processing requests..."
   CGroup: /system.slice/systemd-journald.service
           └─29141 /usr/lib/systemd/systemd-journald
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

Lastly I hope the this article on systemd-journald on Linux was helpful and I was able to give you some insight on how logging works with Journal in RHEL 7 along with systemd-journald logrotate. So, let me know your suggestions and feedback using the comment section.