[HowTo] managing logfiles

Share your HowTo, Documentation, Tips and Tricks. Not for support questions!.
Post Reply
Message
Author
User avatar
wizard10000
Global Moderator
Global Moderator
Posts: 1032
Joined: 2019-04-16 23:15
Location: southeastern us
Has thanked: 114 times
Been thanked: 173 times

[HowTo] managing logfiles

#1 Post by wizard10000 »

I spent some time yesterday cleaning up logfiles and configuring logrotate and journald and figured I'd write a howto on managing logfiles,

Two things we'll get to play with here - logrotate and journald.conf. Note that both logrotate and journald.conf have man pages available in case you get stuck. Let's get to it :)

logrotate

logrotate looks for its configuration in /etc/logrotate.conf and /etc/logrotate.conf.d and the format is pretty simple. Below is an unmolested copy of logrotate.conf - check it out -

Code: Select all

# see "man logrotate" for details.

# global options do not affect preceding include directives

# rotate log files weekly
#weekly

# keep 4 weeks worth of backlogs
#rotate 4

# create new (empty) log files after rotating old ones
#create

# use date as a suffix of the rotated file
#dateext

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# system-specific logs may also be configured here.
Pretty much self-explanatory, right? Now let's look at mine -

Code: Select all

# see "man logrotate" for details

# global options do not affect preceding include directives

# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 2

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
dateext

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d
I've made a couple of changes - I don't need four weeks worth of logs so I've cut that back to two weeks by changing the default "rotate 4" to "rotate 2". I've also told logrotate to create an empty logfile for any log it archives and to add a date to the archive's filename. I don't compress logs but you might want to take advantage of that option.

You can see the last line in the file says "include /etc/logrotate.d" and if you take a look in that directory you'll see configurations for a few other logs - check it out - this is /etc/lograte.d/apt

Code: Select all

/var/log/apt/term.log {
  rotate 2
  weekly
  compress
  missingok
  notifempty
}

/var/log/apt/history.log {
  rotate 2
  weekly
  compress
  missingok
  notifempty
}
Now apt by default rotates logs monthly and keeps a year's worth of logs so as you can see I've made a couple of changes so apt only maintains two weeks of logs. As mentioned logrotate's man page has a lot more information and is worth a look because there's a heck of a lot more stuff you can do with logrotate like run scripts, move archives to another directory and so on. You can create files in /etc/logrotate.d if you want to customize other logfiles.

journald.conf

journald maintains binary logs that can be accessed with journalctl. /etc/systemd/journald.conf is a little more complex than logrotate configs so I've linked to https://www.freedesktop.org/software/sy ... .conf.html so folks can get a better understanding of what goes on under the hood. Check this out -

Code: Select all

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=10000
SystemMaxUse=1G
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
#MaxRetentionSec=0
#MaxFileSec=1month
#ForwardToSyslog=no
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
#MaxLevelSocket=debug
#LineMax=48K
#ReadKMsg=yes
#Audit=yes
If you look closely you'll see the only change I made was restricting the journal to 1GB. By default SystemMaxUse limits the journal to 10% of the filesystem or 4GB, whichever is smaller. I don't need 4GB of journal so I restrict journald to 1GB but I don't restrict it during runtime. If someone is concerned about a runaway log filling up a root partition you could also adjust RuntimeMaxUse to help protect your system.

The changes I made above reduced the size of my logs from a little more than 3GB to 1.2GB. As mentioned logrotate and journald.conf are well-documented but I would recommend backing up the default files before hacking away at them.

Enjoy!

:linked:
we see things not as they are, but as we are.
-- anais nin

User avatar
sunrat
Administrator
Administrator
Posts: 7289
Joined: 2006-08-29 09:12
Location: Melbourne, Australia
Has thanked: 128 times
Been thanked: 636 times

Re: [HowTo] managing logfiles

#2 Post by sunrat »

I restrict journal logs to 200M as I've never found the need for journal logs past the previous boot. YMMV.

Code: Select all

SystemMaxUse=200M
I do however leave apt logs at the default 12 which are archived monthly. On a few occasions I've wanted to find something that was installed long ago. My /var/log/apt/ directory currently amasses 394KiB so that's not much burden.
“ computer users can be divided into 2 categories:
Those who have lost data
...and those who have not lost data YET ”
Remember to BACKUP!

lindi
Debian Developer
Debian Developer
Posts: 556
Joined: 2022-07-12 14:10
Has thanked: 2 times
Been thanked: 110 times

Re: [HowTo] managing logfiles

#3 Post by lindi »

With systemd-journal-remote you can configure your systems to sync the journal to a remote system.

User avatar
wizard10000
Global Moderator
Global Moderator
Posts: 1032
Joined: 2019-04-16 23:15
Location: southeastern us
Has thanked: 114 times
Been thanked: 173 times

Re: [HowTo] managing logfiles

#4 Post by wizard10000 »

sunrat wrote: 2024-10-23 14:09I restrict journal logs to 200M as I've never found the need for journal logs past the previous boot. YMMV.

Code: Select all

SystemMaxUse=200M
Me neither - I may shrink this further.
sunrat wrote: 2024-10-23 14:09I do however leave apt logs at the default 12 which are archived monthly. On a few occasions I've wanted to find something that was installed long ago. My /var/log/apt/ directory currently amasses 394KiB so that's not much burden.
Now that I think about it I'm beginning to agree with you here. I think I may put this back where it was and just shove the 11 archives it'll create into their own directory. I don't care much for clutter and am thinking of creating /var/log/archives and telling journald and logrotate to move archives there.

Thanks for the input!

edit: here's where I ended up -

Code: Select all

/var/log/apt/term.log {
  rotate 12
  monthly
  olddir /var/log/archives/apt
  dateext
  missingok
  notifempty
}

/var/log/apt/history.log {
  rotate 12
  monthly
  olddir /var/log/archives/apt
  dateext
  missingok
  notifempty
}
we see things not as they are, but as we are.
-- anais nin

CwF
Global Moderator
Global Moderator
Posts: 3045
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 62 times
Been thanked: 251 times

Re: [HowTo] managing logfiles

#5 Post by CwF »

For users to manipulate and fully utilize the journal remember to add those users to group ‘adm’ and/or ‘systemd-journal’. This also gives the user the ability to use the journalctl --vacuum.

It's possible on some system to remove rsyslog and logrotate packages since they are redundant. I have not completed a full audit of legacy programs that may still only use the old ways, but the journal is now capturing most things. apt/history would still be one reason to keep logrotate and on a few systems and I have that history years long and not really an issue.

So far I use only the file number limit and/or use active user management during the session.

Code: Select all

 
 $  cat /etc/systemd/journald.conf.d/MaxFiles.conf
SystemMaxFiles=11
 $  journalctl --disk-usage
Archived and active journals take up 62.3M in the file system.
 $  journalctl --vacuum-files=10
Vacuuming done, freed 0B of archived journals from /run/log/journal.
Vacuuming done, freed 0B of archived journals from /var/log/journal.
Vacuuming done, freed 0B of archived journals from /var/log/journal/7f00b65437ca41578997408626d105f8
 
--vacuum-size=BYTES
--vacuum-files=INT
--vacuum-time=TIME

If there is cruft existing before any declarations are made then we should manually vacuum them down to size to see immediate results. Old log files need manually deleted I think.

On Xfce I use genmon with some of this built in to report, and with a click, vacuum. I wish I could click something like that for my house...Where is that Robin...
Bleachbit also has a journal vacuum built in.
Mottainai

User avatar
wizard10000
Global Moderator
Global Moderator
Posts: 1032
Joined: 2019-04-16 23:15
Location: southeastern us
Has thanked: 114 times
Been thanked: 173 times

Re: [HowTo] managing logfiles

#6 Post by wizard10000 »

CwF wrote: 2024-10-23 18:20...I have not completed a full audit of legacy programs that may still only use the old ways
The list is pretty short. On this machine it's alternatives, apt, aptitude, cups, dpkg, fontconfig and popcon.
we see things not as they are, but as we are.
-- anais nin

DebianFox
Posts: 157
Joined: 2024-05-05 14:11
Has thanked: 21 times
Been thanked: 11 times

Re: [HowTo] managing logfiles

#7 Post by DebianFox »

CwF wrote: 2024-10-23 18:20 For users to manipulate and fully utilize the journal remember to add those users to group ‘adm’ and/or ‘systemd-journal’. This also gives the user the ability to use the journalctl --vacuum.

It's possible on some system to remove rsyslog and logrotate packages since they are redundant. I have not completed a full audit of legacy programs that may still only use the old ways, but the journal is now capturing most things. apt/history would still be one reason to keep logrotate and on a few systems and I have that history years long and not really an issue.
@CwF any suggestion on how to carry out the audit of legacy programs that may still use the old way? It would be instructive to find that out.

CwF
Global Moderator
Global Moderator
Posts: 3045
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 62 times
Been thanked: 251 times

Re: [HowTo] managing logfiles

#8 Post by CwF »

DebianFox wrote: 2024-10-30 10:16 any suggestion on how to carry out the audit of legacy programs that may still use the old way?
Not really, other than a report from cruft_ng on a system with history. Overall, application logging has always been diverse and not strictly contained to /var/log. Reading the config files of logrotate and rsyslog will give some clues of what they handle, as hinted /etc/logrotate.d/ is the list for any particular system. Wizard's short list is longer than mine.

Journalctl is a huge improvement and is now very useful. I've added a click on a genmon panel on systems with many automounts to tell me their status, after hiding them from display with udev. So, “noise Mounted” or “noise Unmounted” blips onto the screen and then fades...

Code: Select all

notify-send "noise $(journalctl -g -.noise -n 1 --no-pager | cut -d " " -f 6)"
Mottainai

DebianFox
Posts: 157
Joined: 2024-05-05 14:11
Has thanked: 21 times
Been thanked: 11 times

Re: [HowTo] managing logfiles

#9 Post by DebianFox »

CwF wrote: 2024-10-30 17:12
DebianFox wrote: 2024-10-30 10:16 any suggestion on how to carry out the audit of legacy programs that may still use the old way?
Not really, other than a report from cruft_ng on a system with history. Overall, application logging has always been diverse and not strictly contained to /var/log. Reading the config files of logrotate and rsyslog will give some clues of what they handle, as hinted /etc/logrotate.d/ is the list for any particular system. Wizard's short list is longer than mine.
Yeah application logging has been way to diverse in Linux. Ideally all logs should be written to /var/log or in its subdirectories. For example X.org writes a lot of its log files inside the Users home directory.
CwF wrote: 2024-10-30 17:12
Journalctl is a huge improvement and is now very useful. I've added a click on a genmon panel on systems with many automounts to tell me their status, after hiding them from display with udev. So, “noise Mounted” or “noise Unmounted” blips onto the screen and then fades...

Code: Select all

notify-send "noise $(journalctl -g -.noise -n 1 --no-pager | cut -d " " -f 6)"
Some would disagree with you on that.
Journalctl writes log files in binary format, hence there are issues that come up when commands like grep/egrep/less are used. This is especially important when the system does not boot up.
Additionally Journalctl is tied at hip with systemd which causes a lot of people to have adverse reaction to it.

User avatar
donald
Debian Developer, Site Admin
Debian Developer, Site Admin
Posts: 1325
Joined: 2021-03-30 20:08
Has thanked: 236 times
Been thanked: 281 times

Re: [HowTo] managing logfiles

#10 Post by donald »

@Best_Threads
Typo perfectionish.


"The advice given above is all good, and just because a new message has appeared it does not mean that a problem has arisen, just that a new gremlin hiding in the hardware has been exposed." - FreewheelinFrank

Post Reply