Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

[Software] journald doesn't seem to be auto vacuuming when running?

Linux Kernel, Network, and Services configuration.
Post Reply
Message
Author
sscotter
Posts: 2
Joined: 2022-12-05 16:43

[Software] journald doesn't seem to be auto vacuuming when running?

#1 Post by sscotter »

Code: Select all

# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 11 (bullseye)
Release:        11
Codename:       bullseye
Hi,

I administer around 40 Debian based virtual machines. About 70% are buster, the balance are bullseye. We're in the process of upgrading those Busters to Bulleye since I noticed earlier this month that Buster was EOL as of Sept 2022.

We monitor our systems for amongst over things disk space and have noticed several have at least 4GB of logs in /var/log/journal. We tend to be quite prudent when it comes to allocating resources to VMs to make the most out of a resource, and also consider things like back ups. We also operate a SEIM which all our servers send their logs to, so storing 4GB or more of logs locally is a bit wasteful.

I've set SystemMaxUse=500M in /etc/systemd/journald.conf and restarted the service by executing systemctl status systemd-journald.service, which immediately vaccumed the logs to 500mb.. however a few days later the logs are back up to 4GB.

Example...
root@SERVERNAME:~# systemctl status systemd-journald.service
● systemd-journald.service - Journal Service
     Loaded: loaded (/lib/systemd/system/systemd-journald.service; static)
     Active: active (running) since Mon 2022-12-05 16:39:12 UTC; 20s ago
TriggeredBy: ● systemd-journald-dev-log.socket
             ● systemd-journald-audit.socket
             ● systemd-journald.socket
       Docs: man:systemd-journald.service(8)
             man:journald.conf(5)
   Main PID: 572900 (systemd-journal)
     Status: "Processing requests..."
      Tasks: 1 (limit: 9529)
     Memory: 1.4M
        CPU: 477ms
     CGroup: /system.slice/systemd-journald.service
             └─572900 /lib/systemd/systemd-journald

Dec 05 16:39:12 SERVERNAME systemd-journald[572900]: Journal started
Dec 05 16:39:12 SERVERNAME systemd-journald[572900]: System Journal (/var/log/journal/71b8a1315af275875e623e775d680544) is 488.0M, max 500.0M, 11.9M free.
Dec 05 16:39:12 SERVERNAME systemd-journald[572900]: System Journal (/var/log/journal/71b8a1315af275875e623e775d680544) is 488.0M, max 500.0M, 11.9M free.

Obviously journald is recognising I've set a hard limit of 500mb because it is vacuuming at start up.. so I can only conclude journald isn't performing it's auto vacuuming when running.

I've reviewed https://manpages.debian.org/testing/sys ... .5.en.html but have been unable to discover why SystemMaxUse=500M isn't being honoured.

I have only noticed this on the Bulleye machines but I don't have enough statistics to confirm it's specifically a Bullseye problem.

Any help gratefully received

Cheers

Steve

sscotter
Posts: 2
Joined: 2022-12-05 16:43

Re: [Software] journald doesn't seem to be auto vacuuming when running?

#2 Post by sscotter »

The last time I touched this server was Nov 28th when I set SystemMaxUse=500M in /etc/systemd/journald.conf and restarted the service. You can see from the graph below in doing so I reclaimed 3.5GB of space.. but you can also see it's been steadily climbing back up to 4GB (which is the upper limit SystemMaxUse is able to be set to according to the man page) ever since.

Image

User avatar
sunrat
Administrator
Administrator
Posts: 6412
Joined: 2006-08-29 09:12
Location: Melbourne, Australia
Has thanked: 116 times
Been thanked: 462 times

Re: [Software] journald doesn't seem to be auto vacuuming when running?

#3 Post by sunrat »

I'm no expert but I think SystemMaxUse= sets the limit for active files. You could try reducing the number of archived files by setting SystemMaxFiles= to less than the default of 100.
You could also reduce the amount of logging by setting MaxLevelStore= to say 4 instead of the default of 7 (debug) :-
Controls the maximum log level of messages that are stored in the journal, forwarded to syslog, kmsg, the console or wall (if that is enabled, see above). As argument, takes one of "emerg", "alert", "crit", "err", "warning", "notice", "info", "debug", or integer values in the range of 0–7 (corresponding to the same levels). Messages equal or below the log level specified are stored/forwarded, messages above are dropped. Defaults to "debug" for MaxLevelStore= and MaxLevelSyslog=, to ensure that the all messages are stored in the journal and forwarded to syslog.
“ computer users can be divided into 2 categories:
Those who have lost data
...and those who have not lost data YET ”
Remember to BACKUP!

Post Reply