question about "SystemMaxUse" in " journald.conf"

Kernels & Hardware, configuring network, installing services

question about "SystemMaxUse" in " journald.conf"

Postby Soul Singin' » 2019-11-16 07:44

Hi Everyone!

I'm a newbie here. Could someone please help me understand the documentation that I'm reading?

man journald.conf wrote:SystemMaxUse= and RuntimeMaxUse= control how much disk space the journal may use up at most.

According to ArchWiki, setting:

Code: Select all
SystemMaxUse=50M

in /etc/systemd/journald.conf will limit the "maximum size of the persistent journal." And ArchWiki further states that it's necessary to restart the service for the settings to take effect:

Code: Select all
systemctl restart systemd-journald.service


My question: . Do I understand correctly that that setting will automatically perform a "vacuuming operation" which deletes old journals and thus prevents the journaling from consuming more than 50MB?

If not, what am I not understanding? This documentation is all Greek to me.

Thanks in advance,
- Soul
User avatar
Soul Singin'
 
Posts: 1583
Joined: 2008-12-21 07:02

Re: question about "SystemMaxUse" in " journald.conf"

Postby Head_on_a_Stick » 2019-11-16 19:53

Soul Singin' wrote:Do I understand correctly that that setting will automatically perform a "vacuuming operation" which deletes old journals and thus prevents the journaling from consuming more than 50MB?

Yes, pretty much.

That setting is irrelevant in a stock Debian system though because persistent journaling is set to auto by default and /var/log/journal does not exist so there are no persistent logs at all.
Don't break DebianHow to report bugs

SharpBang GNU/Linux® — a pre-configured Openbox/Tint2 desktop running on Debian stable
User avatar
Head_on_a_Stick
 
Posts: 10695
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: question about "SystemMaxUse" in " journald.conf"

Postby sunrat » 2019-11-16 23:39

I'm a newbie too and this question triggered more questions in my mind. If /var/log/journal is not created, where is all my journal data stored?
I vaguely recall persistent journal can be enabled by simply creating /var/log/journal

Code: Select all
$ journalctl
-- Logs begin at Fri 2019-11-15 11:15:59 AEDT, end at Sun 2019-11-17 10:46:01 AEDT. --
...

Code: Select all
$ journalctl --disk-usage
Archived and active journals take up 9.8M in the file system.


Turns out my 9.8MB of journal is in /run/log:
Code: Select all
/run/log$ du .
10080   ./journal/2b2f2589e1c943e5878f76178a331643

but no journal data is retained after reboot unless "auto" is set for "Storage" in /etc/systemd/journald.conf and /var/log/journal exists, or "persistent' is set for "Storage".

Some good info is at - https://askubuntu.com/questions/864722/ ... ata-stored

Thanks for the interesting question Mr. Singin', and interesting answer Mr. Stick!
“ computer users can be divided into 2 categories:
Those who have lost data
...and those who have not lost data YET ”
Remember to BACKUP!
User avatar
sunrat
 
Posts: 2920
Joined: 2006-08-29 09:12
Location: Melbourne, Australia

Re: question about "SystemMaxUse" in " journald.conf"

Postby Head_on_a_Stick » 2019-11-16 23:51

sunrat wrote:If /var/log/journal is not created, where is all my journal data stored?

Under /run/log/journal, as explained in journald.conf(5).
Don't break DebianHow to report bugs

SharpBang GNU/Linux® — a pre-configured Openbox/Tint2 desktop running on Debian stable
User avatar
Head_on_a_Stick
 
Posts: 10695
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: question about "SystemMaxUse" in " journald.conf"

Postby sunrat » 2019-11-17 00:06

Head_on_a_Stick wrote:
sunrat wrote:If /var/log/journal is not created, where is all my journal data stored?

Under /run/log/journal, as explained in journald.conf(5).


Yes I found that and mentioned it in my post. This thread has prompted me to enable persistent journaling as I recall tracking down at least a couple of issues in the past with
Code: Select all
journalctl -b -1
which shows nothing without persistent journal.
“ computer users can be divided into 2 categories:
Those who have lost data
...and those who have not lost data YET ”
Remember to BACKUP!
User avatar
sunrat
 
Posts: 2920
Joined: 2006-08-29 09:12
Location: Melbourne, Australia

Re: question about "SystemMaxUse" in " journald.conf"

Postby Soul Singin' » 2019-11-17 00:36

Thank you for your responses.

sunrat wrote:I recall tracking down at least a couple of issues in the past with
Code: Select all
journalctl -b -1
which shows nothing without persistent journal.

Me too. That's why I had enabled it.

But those damn journals consumed all of the remaining space in my /(root) partition yesterday.

I had no idea what was going on. All I knew was that applications were misbehaving. LibreOffice, for example, informed me that all of my documents were corrupt. Frightened and not knowing what to do, I tried rebooting the computer, but after the reboot, my window manager could not start an X session.

Finally, I logged at TTY1, ran df -h, noticed that the partition was full and guessed (correctly) that it was the journals. After removing a bunch of those things, my system began operating normally again.

My immediate next step was to limit those buggers to 100MB, but:

Code: Select all
$ journalctl --disk-usage
Archived and active journals take up 112.1M in the file system.
$
$ du -h /var/log/journal/be893b985e5a4053a10312a84fd7c509/
113M    /var/log/journal/be893b985e5a4053a10312a84fd7c509/

Over the limit already. Lovely! . :(

man journald.conf wrote:Also note that only archived files are deleted to reduce the space occupied by journal files. This means that, in effect, there might still be more space used than SystemMaxUse= or RuntimeMaxUse= limit after a vacuuming operation is complete.

If those things screw up my system again, I'm going to turn off persistent journaling. That's a simple matter of deleting the /var/log/journal/ directory, correct?

Thanks for your help!
- Soul
User avatar
Soul Singin'
 
Posts: 1583
Joined: 2008-12-21 07:02

Re: question about "SystemMaxUse" in " journald.conf"

Postby sunrat » 2019-11-17 00:59

Soul Singin' wrote:But those damn journals consumed all of the remaining space in my /(root) partition yesterday.

Happened to me a couple of times with sys.log and kernel.log in pre-systemd days.
Over the limit already. Lovely! . :(

I use siduction rather than pure Debian, but there are 2 files in /etc/systemd/journal.conf.d/ called level.conf and size.conf which override settings in journald.conf. size.conf has contents:
Code: Select all
[Journal]
SystemMaxUse=25M
SystemMaxFileSize=2M

Not sure if you may have them in pure Debian or they are siduction tweaks.

Soul Singin' wrote:If those things screw up my system again, I'm going to turn off persistent journaling. That's a simple matter of deleting the /var/log/journal/ directory, correct?

Yeah if Storage=auto is set in journald.conf that's all you have to do. If you changed it to Storage=persistent, it will recreate /var/log/journal by itself so you need to change it back to "auto" or one of the other less useful options.
“ computer users can be divided into 2 categories:
Those who have lost data
...and those who have not lost data YET ”
Remember to BACKUP!
User avatar
sunrat
 
Posts: 2920
Joined: 2006-08-29 09:12
Location: Melbourne, Australia

Re: question about "SystemMaxUse" in " journald.conf"

Postby Soul Singin' » 2019-11-17 02:39

sunrat wrote:I use siduction rather than pure Debian, but there are 2 files in /etc/systemd/journal.conf.d/ called level.conf and size.conf which override settings in journald.conf. size.conf has contents:
Code: Select all
[Journal]
SystemMaxUse=25M
SystemMaxFileSize=2M

Not sure if you may have them in pure Debian or they are siduction tweaks.

They are not in pure Debian, but perhaps they should be?

man journald.conf wrote:SystemMaxFileSize= and RuntimeMaxFileSize= control how large individual journal files may grow at most. This influences the granularity in which disk space is made available through rotation, i.e. deletion of historic data. Defaults to one eighth of the values configured with SystemMaxUse= and RuntimeMaxUse=, so that usually seven rotated journal files are kept as history.


sunrat wrote:
Soul Singin' wrote:If those things screw up my system again, I'm going to turn off persistent journaling. That's a simple matter of deleting the /var/log/journal/ directory, correct?

Yeah if Storage=auto is set in journald.conf that's all you have to do. If you changed it to Storage=persistent, it will recreate /var/log/journal by itself so you need to change it back to "auto" or one of the other less useful options.

Thank you.

With the help that you and Head_on_a_Stick have provided, hopefully my /(root) partition will never fill up again. Thank you!
User avatar
Soul Singin'
 
Posts: 1583
Joined: 2008-12-21 07:02

Re: question about "SystemMaxUse" in " journald.conf"

Postby Head_on_a_Stick » 2019-11-17 18:07

Soul Singin' wrote:With the help that you and Head_on_a_Stick have provided, hopefully my /(root) partition will never fill up again.

Erm, do you not think it would be a good idea to investigate why your logs are filling up at such an alarming rate rather than just clean them out periodically?
Don't break DebianHow to report bugs

SharpBang GNU/Linux® — a pre-configured Openbox/Tint2 desktop running on Debian stable
User avatar
Head_on_a_Stick
 
Posts: 10695
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: question about "SystemMaxUse" in " journald.conf"

Postby Soul Singin' » 2019-11-17 21:51

Head_on_a_Stick wrote:Erm, do you not think it would be a good idea to investigate why your logs are filling up at such an alarming rate rather than just clean them out periodically?

Erm, yes!

Code: Select all
$ journalctl -r -b 1
-- Logs begin at Sun 2019-11-17 01:27:31 EST, end at Sun 2019-11-17 17:24:10 EST. --
Nov 17 17:24:10 inspiron systemd[1]: expressvpn.service: Failed with result 'exit-code'.
Nov 17 17:24:10 inspiron systemd[1]: expressvpn.service: Main process exited, code=exited, status=203/EXEC
Nov 17 17:24:10 inspiron systemd[8888]: expressvpn.service: Failed at step EXEC spawning /usr/sbin/expressvpnd: No such file or direct
Nov 17 17:24:10 inspiron systemd[8888]: expressvpn.service: Failed to execute command: No such file or directory
Nov 17 17:24:10 inspiron systemd[1]: Started ExpressVPN Daemon.
Nov 17 17:24:10 inspiron systemd[1]: Stopped ExpressVPN Daemon.
Nov 17 17:24:10 inspiron systemd[1]: expressvpn.service: Scheduled restart job, restart counter is at 14812.
Nov 17 17:24:10 inspiron systemd[1]: expressvpn.service: Service RestartSec=5s expired, scheduling restart.
Nov 17 17:24:05 inspiron systemd[1]: expressvpn.service: Failed with result 'exit-code'.
Nov 17 17:24:05 inspiron systemd[1]: expressvpn.service: Main process exited, code=exited, status=203/EXEC
Nov 17 17:24:05 inspiron systemd[8886]: expressvpn.service: Failed at step EXEC spawning /usr/sbin/expressvpnd: No such file or direct
Nov 17 17:24:05 inspiron systemd[8886]: expressvpn.service: Failed to execute command: No such file or directory
Nov 17 17:24:05 inspiron systemd[1]: Started ExpressVPN Daemon.
Nov 17 17:24:05 inspiron systemd[1]: Stopped ExpressVPN Daemon.
Nov 17 17:24:05 inspiron systemd[1]: expressvpn.service: Scheduled restart job, restart counter is at 14811.
Nov 17 17:24:05 inspiron systemd[1]: expressvpn.service: Service RestartSec=5s expired, scheduling restart.
Nov 17 17:24:00 inspiron systemd[1]: expressvpn.service: Failed with result 'exit-code'.
Nov 17 17:24:00 inspiron systemd[1]: expressvpn.service: Main process exited, code=exited, status=203/EXEC
Nov 17 17:24:00 inspiron systemd[8885]: expressvpn.service: Failed at step EXEC spawning /usr/sbin/expressvpnd: No such file or direct
Nov 17 17:24:00 inspiron systemd[8885]: expressvpn.service: Failed to execute command: No such file or directory
Nov 17 17:24:00 inspiron systemd[1]: Started ExpressVPN Daemon.
Nov 17 17:24:00 inspiron systemd[1]: Stopped ExpressVPN Daemon.
Nov 17 17:24:00 inspiron systemd[1]: expressvpn.service: Scheduled restart job, restart counter is at 14810.
Nov 17 17:24:00 inspiron systemd[1]: expressvpn.service: Service RestartSec=5s expired, scheduling restart.
Nov 17 17:23:54 inspiron systemd[1]: expressvpn.service: Failed with result 'exit-code'.
Nov 17 17:23:54 inspiron systemd[1]: expressvpn.service: Main process exited, code=exited, status=203/EXEC

Wild guess here, but I think it might be expressvpn.service.

I removed that package from my system a long time ago, but I never ran:

Code: Select all
# systemctl stop expressvpn.service
# systemctl disable expressvpn.service
Removed /etc/systemd/system/multi-user.target.wants/expressvpn.service.
# rm /etc/systemd/system/expressvpn.service
# systemctl daemon-reload
# systemctl reset-failed

Until now.

Thanks! . :D
- Soul
User avatar
Soul Singin'
 
Posts: 1583
Joined: 2008-12-21 07:02


Return to System configuration

Who is online

Users browsing this forum: No registered users and 14 guests

fashionable