question about "SystemMaxUse" in " journald.conf"

[Soul Singin']

Hi Everyone!

I'm a newbie here. Could someone please help me understand the documentation that I'm reading?

SystemMaxUse= and RuntimeMaxUse= control how much disk space the journal may use up at most.

According to ArchWiki, setting:

Code: Select all

SystemMaxUse=50M

in /etc/systemd/journald.conf will limit the "maximum size of the persistent journal." And ArchWiki further states that it's necessary to restart the service for the settings to take effect:

Code: Select all

systemctl restart systemd-journald.service

My question: . Do I understand correctly that that setting will automatically perform a "vacuuming operation" which deletes old journals and thus prevents the journaling from consuming more than 50MB?

If not, what am I not understanding? This documentation is all Greek to me.

Thanks in advance,
- Soul

[Head_on_a_Stick]

Do I understand correctly that that setting will automatically perform a "vacuuming operation" which deletes old journals and thus prevents the journaling from consuming more than 50MB?

Yes, pretty much.

That setting is irrelevant in a stock Debian system though because persistent journaling is set to auto by default and /var/log/journal does not exist so there are no persistent logs at all.

[sunrat]

I'm a newbie too and this question triggered more questions in my mind. If /var/log/journal is not created, where is all my journal data stored?
I vaguely recall persistent journal can be enabled by simply creating /var/log/journal

Code: Select all

$ journalctl
-- Logs begin at Fri 2019-11-15 11:15:59 AEDT, end at Sun 2019-11-17 10:46:01 AEDT. --
...

Code: Select all

$ journalctl --disk-usage
Archived and active journals take up 9.8M in the file system.

Turns out my 9.8MB of journal is in /run/log:

Code: Select all

/run/log$ du .
10080   ./journal/2b2f2589e1c943e5878f76178a331643

but no journal data is retained after reboot unless "auto" is set for "Storage" in /etc/systemd/journald.conf and /var/log/journal exists, or "persistent' is set for "Storage".

Some good info is at - https://askubuntu.com/questions/864722/ ... ata-stored

Thanks for the interesting question Mr. Singin', and interesting answer Mr. Stick!

[Head_on_a_Stick]

If /var/log/journal is not created, where is all my journal data stored?

Under /run/log/journal, as explained in journald.conf(5).

[sunrat]

If /var/log/journal is not created, where is all my journal data stored?

Under /run/log/journal, as explained in journald.conf(5).

Yes I found that and mentioned it in my post. This thread has prompted me to enable persistent journaling as I recall tracking down at least a couple of issues in the past with

Code: Select all

journalctl -b -1

which shows nothing without persistent journal.

[Soul Singin']

Thank you for your responses.

I recall tracking down at least a couple of issues in the past with

Code: Select all

journalctl -b -1

which shows nothing without persistent journal.

Me too. That's why I had enabled it.

But those damn journals consumed all of the remaining space in my /(root) partition yesterday.

I had no idea what was going on. All I knew was that applications were misbehaving. LibreOffice, for example, informed me that all of my documents were corrupt. Frightened and not knowing what to do, I tried rebooting the computer, but after the reboot, my window manager could not start an X session.

Finally, I logged at TTY1, ran df -h, noticed that the partition was full and guessed (correctly) that it was the journals. After removing a bunch of those things, my system began operating normally again.

My immediate next step was to limit those buggers to 100MB, but:

Code: Select all

$ journalctl --disk-usage
Archived and active journals take up 112.1M in the file system.
$ 
$ du -h /var/log/journal/be893b985e5a4053a10312a84fd7c509/
113M    /var/log/journal/be893b985e5a4053a10312a84fd7c509/

Over the limit already. Lovely! .

Also note that only archived files are deleted to reduce the space occupied by journal files. This means that, in effect, there might still be more space used than SystemMaxUse= or RuntimeMaxUse= limit after a vacuuming operation is complete.

If those things screw up my system again, I'm going to turn off persistent journaling. That's a simple matter of deleting the /var/log/journal/ directory, correct?

Thanks for your help!
- Soul

[sunrat]

But those damn journals consumed all of the remaining space in my /(root) partition yesterday.

Happened to me a couple of times with sys.log and kernel.log in pre-systemd days.

Over the limit already. Lovely! .

I use siduction rather than pure Debian, but there are 2 files in /etc/systemd/journal.conf.d/ called level.conf and size.conf which override settings in journald.conf. size.conf has contents:

Code: Select all

[Journal]
SystemMaxUse=25M
SystemMaxFileSize=2M

Not sure if you may have them in pure Debian or they are siduction tweaks.

If those things screw up my system again, I'm going to turn off persistent journaling. That's a simple matter of deleting the /var/log/journal/ directory, correct?

Yeah if Storage=auto is set in journald.conf that's all you have to do. If you changed it to Storage=persistent, it will recreate /var/log/journal by itself so you need to change it back to "auto" or one of the other less useful options.

[Soul Singin']

I use siduction rather than pure Debian, but there are 2 files in /etc/systemd/journal.conf.d/ called level.conf and size.conf which override settings in journald.conf. size.conf has contents:

Code: Select all

[Journal]
SystemMaxUse=25M
SystemMaxFileSize=2M

Not sure if you may have them in pure Debian or they are siduction tweaks.

They are not in pure Debian, but perhaps they should be?

SystemMaxFileSize= and RuntimeMaxFileSize= control how large individual journal files may grow at most. This influences the granularity in which disk space is made available through rotation, i.e. deletion of historic data. Defaults to one eighth of the values configured with SystemMaxUse= and RuntimeMaxUse=, so that usually seven rotated journal files are kept as history.

If those things screw up my system again, I'm going to turn off persistent journaling. That's a simple matter of deleting the /var/log/journal/ directory, correct?

Yeah if Storage=auto is set in journald.conf that's all you have to do. If you changed it to Storage=persistent, it will recreate /var/log/journal by itself so you need to change it back to "auto" or one of the other less useful options.

Thank you.

With the help that you and Head_on_a_Stick have provided, hopefully my /(root) partition will never fill up again. Thank you!

[Head_on_a_Stick]

With the help that you and Head_on_a_Stick have provided, hopefully my /(root) partition will never fill up again.

Erm, do you not think it would be a good idea to investigate why your logs are filling up at such an alarming rate rather than just clean them out periodically?

[Soul Singin']

Erm, do you not think it would be a good idea to investigate why your logs are filling up at such an alarming rate rather than just clean them out periodically?

Erm, yes!

Code: Select all

$ journalctl -r -b 1
-- Logs begin at Sun 2019-11-17 01:27:31 EST, end at Sun 2019-11-17 17:24:10 EST. --
Nov 17 17:24:10 inspiron systemd[1]: expressvpn.service: Failed with result 'exit-code'.
Nov 17 17:24:10 inspiron systemd[1]: expressvpn.service: Main process exited, code=exited, status=203/EXEC
Nov 17 17:24:10 inspiron systemd[8888]: expressvpn.service: Failed at step EXEC spawning /usr/sbin/expressvpnd: No such file or direct
Nov 17 17:24:10 inspiron systemd[8888]: expressvpn.service: Failed to execute command: No such file or directory
Nov 17 17:24:10 inspiron systemd[1]: Started ExpressVPN Daemon.
Nov 17 17:24:10 inspiron systemd[1]: Stopped ExpressVPN Daemon.
Nov 17 17:24:10 inspiron systemd[1]: expressvpn.service: Scheduled restart job, restart counter is at 14812.
Nov 17 17:24:10 inspiron systemd[1]: expressvpn.service: Service RestartSec=5s expired, scheduling restart.
Nov 17 17:24:05 inspiron systemd[1]: expressvpn.service: Failed with result 'exit-code'.
Nov 17 17:24:05 inspiron systemd[1]: expressvpn.service: Main process exited, code=exited, status=203/EXEC
Nov 17 17:24:05 inspiron systemd[8886]: expressvpn.service: Failed at step EXEC spawning /usr/sbin/expressvpnd: No such file or direct
Nov 17 17:24:05 inspiron systemd[8886]: expressvpn.service: Failed to execute command: No such file or directory
Nov 17 17:24:05 inspiron systemd[1]: Started ExpressVPN Daemon.
Nov 17 17:24:05 inspiron systemd[1]: Stopped ExpressVPN Daemon.
Nov 17 17:24:05 inspiron systemd[1]: expressvpn.service: Scheduled restart job, restart counter is at 14811.
Nov 17 17:24:05 inspiron systemd[1]: expressvpn.service: Service RestartSec=5s expired, scheduling restart.
Nov 17 17:24:00 inspiron systemd[1]: expressvpn.service: Failed with result 'exit-code'.
Nov 17 17:24:00 inspiron systemd[1]: expressvpn.service: Main process exited, code=exited, status=203/EXEC
Nov 17 17:24:00 inspiron systemd[8885]: expressvpn.service: Failed at step EXEC spawning /usr/sbin/expressvpnd: No such file or direct
Nov 17 17:24:00 inspiron systemd[8885]: expressvpn.service: Failed to execute command: No such file or directory
Nov 17 17:24:00 inspiron systemd[1]: Started ExpressVPN Daemon.
Nov 17 17:24:00 inspiron systemd[1]: Stopped ExpressVPN Daemon.
Nov 17 17:24:00 inspiron systemd[1]: expressvpn.service: Scheduled restart job, restart counter is at 14810.
Nov 17 17:24:00 inspiron systemd[1]: expressvpn.service: Service RestartSec=5s expired, scheduling restart.
Nov 17 17:23:54 inspiron systemd[1]: expressvpn.service: Failed with result 'exit-code'.
Nov 17 17:23:54 inspiron systemd[1]: expressvpn.service: Main process exited, code=exited, status=203/EXEC

Wild guess here, but I think it might be expressvpn.service.

I removed that package from my system a long time ago, but I never ran:

Code: Select all

# systemctl stop expressvpn.service
# systemctl disable expressvpn.service
Removed /etc/systemd/system/multi-user.target.wants/expressvpn.service.
# rm /etc/systemd/system/expressvpn.service
# systemctl daemon-reload
# systemctl reset-failed

Until now.

Thanks! .
- Soul