[Solved] journald’s storage: first it was volatile, then persistent, then... back to volatile?

[yellowfinch]

From what I read in this forum and others: up to and including Debian 10, journald was writing to /run/log/. Then with Debian 11 (starting in 2019, or 2020 for the stable branch) there were announcements and discussions about journald’s default being set to persistent storage henceforth. (Not “auto”, but outright persistent. See What's new in Debian 11.) I didn’t really find anything on to this topic beyond the year 2020 or in relation to Debian 12.

I suppose the change to a persistent log must have stuck for those who somehow upgraded their Debian 11 to Debian 12.

But what if one does a fresh install of Debian 12? That’s what I did. (I had nothing to upgrade because I had never used Debian before February 2024. I was on Linux Mint before; it's based on Ubuntu, but behind by a few years.)

And what I found in Debian 12 is that the log is not persistent. The Storage= parameter is followed by auto but the line is commented out! Was persistence really the default only in Debian 11?

Anyway...

I need persistence, at least for a while.

So I un-commented the Storage line in /etc/systemd/journald.conf, changed its option from auto to persistent and restarted systemd-journald.service. (Also systemd-journald at first, because that’s the instruction I had found initially.)

But the log still gets created in /run/log/.

Even rebooting doesn’t cause the new option to take effect.

What am I missing?

[CwF]

I can't comment on default 11 or a fresh 12! When I run across it, I fix it, then forget...

All my mixes are upgrades and all seem to be the same;
/run/log/journal/ is empty
/var/log/journal/ holds the persistent logs
/etc/systemd/journald.conf default settings (no changes)
/etc/systemd/journald.conf.d/override.conf are user created settings

Comb through the options and create your preferences in /etc/systemd/journald.conf.d/
man journald.conf.d

Some still have older log packages like rsyslog, some don't, but that seems parallel.

[lindi]

And what I found in Debian 12 is that the log is not persistent. The Storage= parameter is followed by auto but the line is commented out! Was persistence really the default only in Debian 11?

If I install Debian 12 all lines in /etc/systemd/journald.conf are commented out but the log is persistent. If I reboot I can access old log entries with journalctl.

[yellowfinch]

Comb through the options and create your preferences in /etc/systemd/journald.conf.d/
man journald.conf.d

Thanks! Copying the journald.conf file to /etc/systemd/journald.conf.d/ (and rebooting) solved it.

[sunrat]

From man journald.conf

Storage= ...
"auto" behaves like "persistent" if the /var/log/journal directory exists, and "volatile" otherwise (the existence of the directory controls the storage mode)...

Default is "auto". Read the man for more info.

[yellowfinch]

I spoke too soon.

Copying journald.conf to /etc/systemd/journald.conf.d/ had the effect of creating the journal files in /var/log/journal/ but there is still no persistence despite the presence of the Storage=persistent parameter, as well as SystemMaxFiles=5. I still get only the one journal file created during the last bootup.

[CwF]

presence of the Storage=persistent

Remove that and try again. As @sunrat suggest, now the directory exist.

[yellowfinch]

presence of the Storage=persistent

Remove that and try again. As @sunrat suggest, now the directory exist.

I did. It doesn't make any difference.

[sunrat]

What does this show?

Code: Select all

ls /var/log/journal/ 

[CwF]

Code: Select all

$  journalctl --field=JOURNAL_PATH
/var/log/journal/e6f789d507d24f80a0f52e07a1e6eb31
/run/log/journal/e6f789d507d24f80a0f52e07a1e6eb31

? Not sure where to look?
I looked at some of the variables and see nothing to confirm or deny 'persistence'
Are there any other log related utilities installed?

All of my upgrades needed the logging reigned in, none needed to be turned on.

[yellowfinch]

Are there any other log related utilities installed?

Nope. This is a fairly recent (three months), plain (i.e. default) installation of bookworm-stable.

Code: Select all

sudo journalctl --field=JOURNAL_PATH
/var/log/journal/fba949a082f6471996bc1d2aed0b0295
/run/log/journal/fba949a082f6471996bc1d2aed0b0295

And the first of those directories contains a file named user-1000.journal plus a symlink named system.journal which simply points to the file.

The fact that there is a file whose name is my non-root user indicates that journald is operating in “split mode”, which is the default, but which the journald.conf manpage says “is only available for journals stored persistently”. Therefore, it looks like journald considers itself to be in persistent mode but something else is destroying the file at bootup. (Or upon shutdown.)

And that’s when I suddenly remembered: Unlike Linux Mint, Debian’s default (at least since version 12, since it’s the first one I’ve used) is to have /tmp/ redirected to a RAM-mounted tmpfs partition and, when I saw that three months ago, I had decided to apply the same principle to /var/log/.

Commenting out the corresponding line in /etc/fstab and rebooting has finally solved the issue:

Code: Select all

sudo journalctl --list-boots
IDX BOOT ID                          FIRST ENTRY                 LAST ENTRY                 
 -2 b53d87b5b54b4031ad3847144b998c95 Mon 2024-02-05 17:35:03 EST Mon 2024-02-05 18:30:57 EST
 -1 8a7e8817345746f2b1af3f2949398ba7 Tue 2024-02-06 09:38:08 EST Tue 2024-02-06 12:23:25 EST
  0 940c86444ea2495689a764eb20d44b32 Tue 2024-05-07 09:24:31 EDT Tue 2024-05-07 09:28:11 EDT

[CwF]

And that’s when I suddenly remembered:

Very good. [reSolved]
I try to note even the smallest tweaks in my CherryTree, I forget too.
I think you can configure it to pull together log locations at shutdown with a --sync if you'd like.

Btw, add your user to 'adm' or 'systemd-journal' to eliminate the sudo usage for reviewing journals. This may allow better filtering concerning user-only entries without root user noise.