How to enable persistent logging in systemd-journald without reboot

In this article I will share the steps to enable persistent logging in systemd-journald without rebooting the node or restarting the systemd-journald service. The systemd journal is configured by default to store logs only in a small ring-buffer in /run/log/journal.

The default storage type in journald.conf is "auto". In auto storage type the journal logs will not be persistent and will not survive reboots.

Different Storage Type supported with journald.conf

• The storage type value is controlled using /etc/systemd/journald.conf file. Under [Journal] you can modify the storage type.
• The supported values are “volatile”, “persistent”, “auto” and “none”
• Default storage type is configured as “auto”
• If “volatile“, journal log data will be stored only in memory, i.e. below the /run/log/journal hierarchy (which is created if needed)
• If “persistent“, data will be stored preferably on disk, i.e. below the /var/log/journal hierarchy, with a fallback to /run/log/journal during early boot stage and if the disk is not writable
• The “auto” value will configure journald to store journal log data in the /var/log/journal/ directory. However, the directory must already exist and have the proper permissions set. If it does not exist, then journal data is stored in the volatile /run/log/journal/ directory, and the data is erased when the system shuts down.
• “none” turns off all storage, all log data received will be dropped.

Enable persistent logging in systemd-journald

Method 1:
Now if your current storage type is auto in /etc/systemd/journald.conf, in that case the “auto” value will configure journald to store journal log data in the /var/log/journal/ directory. However, the directory must already exist and have the proper permissions set. If it does not exist, then journal data is stored in the volatile /run/log/journal/ directory, and the data is erased when the system shuts down.

Now in this case to enable persistent logging in systemd-journald you just need to create a directory /var/log/journal/ and systemd-journald will start writing journal log files into the disk rather than memory.

With this change the logs will become persistent and will not be erased after reboot

Single step to be followed

# mkdir -p /var/log/journal

No need to restart the systemd-journald service or reboot the node. As soon as this directory is accessible, systemd-journald will start writing logs under this path

# ls -l /var/log/journal/3a0d751560f045428773cbf4c1769a5c/
total 8192
-rw-r----- 1 root root 8388608 Sep  4 08:15 system.journal

Method 2:
There is no problem in using the steps from Method 1 but again to enable persistent logging in systemd-journal properly we must change the "Storage" type to "persistent". In which case you need not manually create /var/log/journal directory.

Replace storage type with Storage=persistent

# sed -i 's/#Storage.*/Storage=persistent/' /etc/systemd/journald.conf

Next you can restart systemd-journald service

# systemctl restart systemd-journald.service

With storage option "persistent", data will be stored preferably on disk, i.e. below the /var/log/journal hierarchy (which is created if needed), with a fallback to /run/log/journal (which is created if needed), during early boot and if the disk is not writable.

You can always use flush to move the journal log files from /run/log/journal to /var/log/journal

# journalctl --flush

here this command asks the Journal daemon to flush any log data stored in /run/log/journal into /var/log/journal, if persistent storage is enabled. This call does not return until the operation is complete

Lastly I hope the steps from the article to enable persistent logging in systemd-journald on Linux was helpful. So, let me know your suggestions and feedback using the comment section.