/var/log/syslog (9Gb) full of snmpd logs fills my /var !

[486DX2]

Hello to all and thanks for reading!
------------------------------------

Since July 7, my /var/syslog (and daemon.log) are full of snmpd messages which look like :

Code: Select all

Jul  7 22:08:10 <hostname> snmpd[<PID>]: send response: Failure in sendto (error parsing snmp message version)
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.1.0
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.2.0
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.3.0
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.4.0
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.5.0
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.6.0
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.7.0
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.8.0
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.2.1
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.2.2
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.2.3
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.2.4
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.2.5
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.2.6
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.2.7
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.2.8
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.2.9
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.2.10
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.3.1
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.3.2
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.3.3
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.3.4
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.3.5
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.3.6
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.3.7
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.3.8
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.3.9
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.3.10
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.4.1
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.4.2
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.4.3
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.4.4
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.4.5
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.4.6
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.4.7
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.4.8
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.4.9
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.9.1.4.10
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.25.1.1.0
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.25.1.2.0
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.25.1.3.0
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.25.1.4.0
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.25.1.5.0
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.25.1.6.0
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.25.1.7.0
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.25.1.7.0
Jul  7 22:08:11 <hostname> snmpd[<PID>]: send response: Failure in sendto

and it follows with same OID

Code: Select all

Jul  7 22:08:11 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.1.0
Jul  7 22:08:11 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.2.0
Jul  7 22:08:11 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.3.0
Jul  7 22:08:11 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.4.0
Jul  7 22:08:11 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.5.0
Jul  7 22:08:11 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.6.0
Jul  7 22:08:11 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.7.0
Jul  7 22:08:11 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.1.8.0

until

Code: Select all

Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.25.1.1.0
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.25.1.2.0
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.25.1.3.0
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.25.1.4.0
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.25.1.5.0
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.25.1.6.0
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.25.1.7.0
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.25.1.7.0
Jul  7 22:08:11 <hostname> snmpd[<PID>]: send response: Failure in sendto

0) <hostname> is the same machine that receives the logs.

1) this logs seem appear from 10/07/2024.

2) /etc/snmp/snmpd.conf was not modified since 2022.

3) Under ACCESS CONTROL in /etc/snmp/snmpd.conf i have:

Code: Select all

view   systemonly  included   .1.3.6.1.2.1.1
view   systemonly  included   .1.3.6.1.2.1.25.1

4) Before error message OID iso.3.6.1.2.1.25.1.7.0 appears twice:

Code: Select all

Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.25.1.7.0
Jul  7 22:08:10 <hostname> snmpd[<PID>]:     -- iso.3.6.1.2.1.25.1.7.0
Jul  7 22:08:11 <hostname> snmpd[<PID>]: send response: Failure in sendto

Could someone help me to resolve that symptom which fills (9Gb) two of my logs files ?

I have tried to play with force logrotate but it's just a remedy and does not cure the cause.

Thanks in advance.

Philippe

[fabien]

Hello, welcome to the forums!

Try to spot changes in your system around 10/07/2024. Security updates on this date?

Code: Select all

$> less /var/log/apt/history.log

Note: please use code tags for terminal outputs. I fixed it for you this time.

[486DX2]

Above all, Thanks for your time Fabien!


My last modified .log under /var/log/apt/ is 2023-07-03


I'm always concerning about message in log file :
Jul 7 22:08:10 <hostname> snmpd[<PID>]: send response: Failure in sendto (error parsing snmp message version)

I'm trying to find information about snmpd running on this machine (with a zabbix-agent too) and a line in /etc/snmp/snmpd.conf who says :

rocommunity <priv_xyz> <IP with no ping available>

According you, this IPv4 with no ping response (no "telnet <IP> 161") could generate this error message of "Failure in sendto" ? or should i focus on "error parsing snmp message version" ?



"To extinguish the flames but leave the embers"...

- I can't force a log rotate (partition is full!) or change to cron daily
- cp a new daemon.log.1 (10Ko) to crash daemon.log.1 (9 GB) with daemon.log untouched and daemon.log.1 gzipped on another partition (even if redundant logs) should not be suitable command ?
- restart snmpd daemon would change nothing...


Thank you for your kind attention Fabien
Philippe

[fabien]

According you, this IPv4 with no ping response (no "telnet <IP> 161") could generate this error message of "Failure in sendto" ? or should i focus on "error parsing snmp message version" ?

Unfortunately, I have no experience with your main problem. I can only provide some general advice.

To list files modified since July 9th, e.g. in /etc/, you can use:

Code: Select all

#> find /etc/ -mtime $(( $(LANG="C" date --date="2024-7-9 00:00:00" '+%j') - $(date '+%j') )) -ls

-atime n
File was last accessed less than, more than or exactly n*24 hours ago. When find figures out how many 24-hour periods ago the file was last accessed, any fractional part is ignored, so
to match -atime +1, a file has to have been accessed at least two days ago.
[...]
-mtime n
File's data was last modified less than, more than or exactly n*24 hours ago. See the comments for -atime to understand how rounding affects the interpretation of file modification
times.

My last modified .log under /var/log/apt/ is 2023-07-03

Does this means that your system has not received any updates since that date?

[486DX2]

Thanks for your share fabien

I'm trying to investigate more....


<< My last modified .log under /var/log/apt/ is 2023-07-03 >>
" Does this means that your system has not received any updates since that date?"

Only individual packages installations seem appear in /var/log/apt/history.log

eg

Code: Select all

cat /var/log/apt/history.log
......
Start-Date: 2024-07-18  09:59:38
Commandline: apt install git
Requested-By: XXXXXXXXXXXXXXXX (1000)
Install: git:amd64 (1:2.30.2-1+deb11u2), patch:amd64 (2.7.6-7, automatic), liberror-perl:amd64 (0.17029-1, automatic), git-man:amd64 (1:2.30.2-1+deb11u2, automatic)
End-Date: 2024-07-18  09:59:42

"apt update" doesn't seem to appear there. I'm wondering about "apt upgrade"...

Philippe

[fabien]

"apt update" doesn't seem to appear there. I'm wondering about "apt upgrade"...

It is not normal. Updates (just updating the cache) are not logged, but upgrades are.
Post the output of

Code: Select all

$> apt list --installed "linux-image-*" libc6

using code tags please, I fixed it again for you in your previous post. We are pretty adamant about this here

[486DX2]

Hello Fabien

Code: Select all

root@xxxxxxxx:~# apt list --installed "linux-image-*" libc6
Listing... Done
libc6/oldstable-updates,now 2.31-13+deb11u5 amd64 [installed,upgradable to: 2.31-13+deb11u6]
linux-image-3.16.0-4-amd64/now 3.16.51-3 amd64 [installed,local]
linux-image-4.19.0-23-amd64/now 4.19.269-1 amd64 [installed,local]
linux-image-5.10.0-21-amd64/oldstable-security,now 5.10.162-1 amd64 [installed,automatic]
linux-image-amd64/now 5.10.162-1 amd64 [installed,upgradable to: 5.10.179-1]

Does it meens someting to you (Debian 11.6) ?

Code: Select all

root@xxxxxxxx:~# uname -a
Linux xxxxxxxx 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux

And thanks for your welcomed help Fabien

[RedGreen925]

Code: Select all

root@xxxxxxxx:~# apt list --installed "linux-image-*" libc6

libc6/oldstable-updates,now 2.31-13+deb11u5 amd64 [installed,upgradable to: 2.31-13+deb11u6]
linux-image-amd64/now 5.10.162-1 amd64 [installed,upgradable to: 5.10.179-1]

Does it meens someting to you (Debian 11.6) ?

Code: Select all

root@xxxxxxxx:~# uname -a
Linux xxxxxxxx 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux

The 11.6 means you are running Bullseye. With a kernel that has not been updated in year and a half, the current version is at the 5.10.221-1.

https://packages.debian.org/search?suit ... mage-amd64

The libc6 is behind as well current version is at 2.31-13+deb11u10.

https://packages.debian.org/search?keyw ... ection=all

Both of the places where it says they are upgradable are wrong in the version available to go to. Once you get the log problem sorted, no clue here as to problem, you should be looking at getting the machine updated as by looking at my machine you are a full four point releases behind, the current version is at 11.10.

Code: Select all

zeus@bullseye-raspi:~$ cat /etc/debian_version 
11.10

[fabien]

As @RedGreen925 pointed out, your apt database is not updated. Can you please run

Code: Select all

#> apt update

(please post the output if you see anything abnormal)
and then post the output of

Code: Select all

$> apt list --installed "linux-image-*" libc6 mawk

This is diagnostic of apt's health, but it could also indicate a larger problem if the apt database update fails.

Of course, always check that you have a good amount of free space in /var/ before doing any administrative task involving apt.
Could you please post the output of

Code: Select all

$> df -h

[486DX2]

Above all Thanks to you Fabien and RedGreen925 for your advices!


<< Of course, always check that you have a good amount of free space in /var/ before doing any administrative task involving apt. >>

Of course, not! Due to my talkative (verbose) SNMPD logs... my /var/ is full and logs are stopped.

Code: Select all

/dev/mapper/debian-var              35G   35G     0 100% /var

I've already tried to free disc space but logs are too huged :


/var/log/daemon.log - 7 Gb
/var/log/daemon.log.1 - 9 Gb
/var/log/daemon.log.2.gz - 64 kb
/var/log/daemon.log.3.gz - 43 kb
...

/var/log/syslog - 7 Gb
/var/log/syslog.1 - 9 Gb
/var/log/syslog.2.gz - 167 kb
/var/log/syslog.3.gz - 143 kb
...


Rotation is done "WEEKLY" and "rotate 4" (/etc/logrotate.d/rsyslog). But with a 100% full /var/ logrotate does nothing !

1) I was thinking about to change logrotate to "DAILY" or to add "size 500M" under /etc/logrotate.d/rsyslog to gz redundant logs and free disc space but it's already full!

2) I was thinking about possible Unallocated Space to extend /var/ as "LVM"

3) Could i shoot /var/log/daemon.log.1 (9Gb) with a /var/log/daemon.log.1 (10 Kb) (with same rights) and then change logrotate conf ? I am not sure at all....


All these 3 fixes are not real solutions (find cause of error SNMPD logs) but it could let me some more time to investigate.

What about 3) ?

Again, thanks for your help!

[fabien]

Rotation is done "WEEKLY" and "rotate 4" (/etc/logrotate.d/rsyslog). But with a 100% full /var/ logrotate does nothing !

systemd-journald is also probably running.

Code: Select all

#> systemctl --type=service list-units | grep -i "log\|journal"
  rsyslog.service                                                                           loaded active running System Logging Service
  smartmontools.service                                                                     loaded active running Self Monitoring and Reporting Technology (SMART) Daemon
  systemd-journal-flush.service                                                             loaded active exited  Flush Journal to Persistent Storage
  systemd-journald.service                                                                  loaded active running Journal Service
  systemd-logind.service                                                                    loaded active running User Login Management

Code: Select all

#> systemctl status   systemd-journald.service   rsyslog.service

You should be able to stop them momentarily:

Code: Select all

#> systemctl stop   systemd-journald.service   rsyslog.service

Can you consider creating a separate partition for /var/log/ ? A full /var/ is highly problematic as it can lead to any type of malfunction that can be very difficult, if not impossible, to spot. By placing the logs in their own partition you are almost safe about this.

[486DX2]

Code: Select all

root@xxxxxxxx:~# systemctl --type=service list-units | grep -i "log\|journal"
● logrotate.service                                                                         loaded failed failed  Rotate log files
  rsyslog.service                                                                           loaded active running System Logging Service
  systemd-journal-flush.service                                                             loaded active exited  Flush Journal to Persistent Storage
  systemd-journald.service                                                                  loaded active running Journal Service
  systemd-logind.service                                                                    loaded active running User Login Management

Code: Select all

root@xxxxxxxx:~# systemctl status systemd-journald.service

● systemd-journald.service - Journal Service
     Loaded: loaded (/lib/systemd/system/systemd-journald.service; static)
     Active: active (running) since Thu 2024-04-18 04:03:31 CEST; 3 months 4 days ago
TriggeredBy: ● systemd-journald.socket
             ● systemd-journald-dev-log.socket
             ● systemd-journald-audit.socket
       Docs: man:systemd-journald.service(8)
             man:journald.conf(5)
   Main PID: 291 (systemd-journal)
     Status: "Processing requests..."
      Tasks: 1 (limit: 2357)
     Memory: 21.8M
        CPU: 18h 46min 6.254s
     CGroup: /system.slice/systemd-journald.service
             └─291 /lib/systemd/systemd-journald

Warning: journal has been rotated since unit was started, output may be incomplete.

Code: Select all

● rsyslog.service - System Logging Service
     Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-04-18 04:04:11 CEST; 3 months 4 days ago
TriggeredBy: ● syslog.socket
       Docs: man:rsyslogd(8)
             man:rsyslog.conf(5)
             https://www.rsyslog.com/doc/
   Main PID: 664 (rsyslogd)
      Tasks: 4 (limit: 2357)
     Memory: 3.0M
        CPU: 15h 24min 24.650s
     CGroup: /system.slice/rsyslog.service
             └─664 /usr/sbin/rsyslogd -n -iNONE

Jul 22 15:12:24 xxxxxxxx rsyslogd[664]: file '/var/log/syslog'[8] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No spa>
Jul 22 15:12:24 xxxxxxxx rsyslogd[664]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error>
Jul 22 15:12:24 xxxxxxxx rsyslogd[664]: file '/var/log/daemon.log'[9] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No>
Jul 22 15:12:24 xxxxxxxx rsyslogd[664]: action 'action-2-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error>
Jul 22 15:12:24 xxxxxxxx rsyslogd[664]: file '/var/log/syslog'[8] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No spa>
...

OK. I followed this interesting link : https://www.rsyslog.com/solving-rsyslog-write-errors/ which mention potential causes when "write error" occurs... but mine seems obvious :

<< Did the system (temporarily) run out of space? This could especially be the case for intermittent problems. >>




As the syslog is in error... Could i stop these two services, delete /var/log/syslog* and /var/log/daemon* and then restart services ??

I've tried on a test VM debian and it seems to work...


However,

1) I had to disabled services before to stop them...

Code: Select all

 root@LINUX-TEST:/home/philippe# systemctl stop systemd-journald.service rsyslog.service
Warning: Stopping systemd-journald.service, but it can still be activated by:
  systemd-journald-audit.socket
  systemd-journald.socket
  systemd-journald-dev-log.socket
Warning: Stopping rsyslog.service, but it can still be activated by:
  syslog.socket

Code: Select all

root@LINUX-TEST:/home/philippe# systemctl disable rsyslog.service
root@LINUX-TEST:/home/philippe# systemctl stop rsyslog.service

To get:

Code: Select all

root@LINUX-TEST:/home/philippe# service --status-all
 [ + ]  apparmor
 [ - ]  console-setup.sh
 [ + ]  cron
 [ + ]  dbus
 [ - ]  hwclock.sh
 [ - ]  keyboard-setup.sh
 [ + ]  kmod
 [ + ]  networking
 [ + ]  open-vm-tools
 [ + ]  procps
 [ - ]  rsyslog
 [ + ]  ssh
 [ - ]  sudo
 [ + ]  udev

Logrotate and journald don't appear in this command....... However they are present in /lib/systemd/system/ and in "systemctl status <service>"

So i did

Code: Select all

root@LINUX-TEST:/home/philippe# systemctl disable systemd-journald.service
root@LINUX-TEST:/home/philippe# systemctl stop systemd-journald.service

2) I deleted /var/log/syslog* and /var/log/daemon* and re-enable/restart serviceS (rsyslog and journald) to get fresh 1 kb /var/log/syslog and /var/log/daemon.log...



Could I save my /var/ size partition with this action (stop services -> delete 9 gb logs files -> restart) combined with a new logrotate configuration less permissive (DAILY or size 500 M) ??



(Thanks for the time to read my post !)

[fabien]

Code: Select all

root@LINUX-TEST:/home/philippe# service --status-all
 [ + ]  apparmor
 [ - ]  console-setup.sh
[...]

Logrotate and journald don't appear in this command....... However they are present in /lib/systemd/system/ and in "systemctl status <service>"

services are now managed by systemd, use systemctl.
systemd-journald collects its own logs:

Code: Select all

#> du -sh /var/log/journal/

2) I deleted /var/log/syslog* and /var/log/daemon* and re-enable/restart serviceS (rsyslog and journald) to get fresh 1 kb /var/log/syslog and /var/log/daemon.log...

So, are your logs still spammed with snmpd messages? If so, can you afford a reboot?

Could I save my /var/ size partition with this action (stop services -> delete 9 gb logs files -> restart) combined with a new logrotate configuration less permissive (DAILY or size 500 M) ??

Possibly. You will need to configure rsyslog and systemd-journald. What do you think about creating a dedicated /var/log/ partition?

[486DX2]

Hello!

486DX2 wrote: ↑2024-07-22 17:03
2) I deleted /var/log/syslog* and /var/log/daemon* and re-enable/restart serviceS (rsyslog and journald) to get fresh 1 kb /var/log/syslog and /var/log/daemon.log...

So, are your logs still spammed with snmpd messages? If so, can you afford a reboot?

I've done 2) ... on VM "test/dev" only... (looking for a snapshot first). Yes, logs still spammed with snmpd messages (looking for a snmpd dedicated forums... where i hope to find people like you fabien)

486DX2 wrote: ↑2024-07-22 17:03
Could I save my /var/ size partition with this action (stop services -> delete 9 gb logs files -> restart) combined with a new logrotate configuration less permissive (DAILY or size 500 M) ??

Possibly. You will need to configure rsyslog and systemd-journald. What do you think about creating a dedicated /var/log/ partition?

-What do you mean about "configure rsyslog and systemd-journald" ? I was thinking to change logrotate only about configuration.

Code: Select all

 What do you think about creating a dedicated /var/log/ partition?

- Yes i thing dedicated /var/log/partition is a good idea. But i can't extend eternally /var/log and i would prefer to find the cause of SNMP error message :

Code: Select all

Jul 15 14:52:13 xxxxxxxx snmpd[690]:     -- iso.3.6.1.2.1.25.1.5.0
Jul 15 14:52:13 xxxxxxxx snmpd[690]:     -- iso.3.6.1.2.1.25.1.6.0
Jul 15 14:52:13 xxxxxxxx snmpd[690]:     -- iso.3.6.1.2.1.25.1.7.0
Jul 15 14:52:13 xxxxxxxx snmpd[690]:     -- iso.3.6.1.2.1.25.1.7.0
Jul 15 14:52:13 xxxxxxxx snmpd[690]: send response: Failure in sendto
...

Code: Select all

root@xxxxxxxx:~# du -sh /var/log/journal/
810M    /var/log/journal/

(I have to read more about LVM) but it looks like on HDD /dev/sda i have 2 partitions (/dev/sda4 and /dev/sda5) which allocates 35 Gb for /var/

Code: Select all

root@xxxxxxxx:~# lsblk
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
fd0                         2:0    1    4K  0 disk
sda                         8:0    0  100G  0 disk
├─sda1                      8:1    0  243M  0 part /boot
├─sda2                      8:2    0 1023K  0 part
├─sda3                      8:3    0    1K  0 part
├─sda4                      8:4    0   60G  0 part
│ ├─debian-root           254:2    0    4G  0 lvm  /
│ ├─debian-var            254:4    0   35G  0 lvm  /var
│ ├─debian-tmp            254:6    0  2.1G  0 lvm  /tmp
│ └─debian-home           254:7    0 33.2G  0 lvm  /mnt/yyyyyy
└─sda5                      8:5    0 39.8G  0 part
  ├─debian-root           254:2    0    4G  0 lvm  /
  ├─debian-usr            254:3    0  8.4G  0 lvm  /usr
  ├─debian-var            254:4    0   35G  0 lvm  /var
  ├─debian-swap_1         254:5    0  724M  0 lvm  [SWAP]
  ├─debian-tmp            254:6    0  2.1G  0 lvm  /tmp
  └─debian-home           254:7    0 33.2G  0 lvm  /mnt/yyyyyy

Thanks to share your knowledge!

[fabien]

I've done 2) ... on VM "test/dev" only... (looking for a snapshot first). Yes, logs still spammed with snmpd messages

Is your VM an image of the real system? And the problems occur there too? If not, can you afford a real system reboot?

-What do you mean about "configure rsyslog and systemd-journald" ? I was thinking to change logrotate only about configuration.

Yes, you are right. Take a look at the systemd-journald manual though, search for "Size".
systemd-journald seems to be more resilient to this type of problem. Try starting it while keeping rsyslog stopped and see what happens.

- Yes i thing dedicated /var/log/partition is a good idea. But i can't extend eternally /var/log and i would prefer to find the cause of SNMP error message :

Of course, that's the main goal, finding out the cause.
The idea is not to extend /var/log/ indefinitely, the idea is to protect /var/ which is very important for the system. A problem with /var/ can be the root cause of many problems, including maybe the one at hand.

[486DX2]

486DX2 wrote: ↑2024-07-23 10:09
I've done 2) ... on VM "test/dev" only... (looking for a snapshot first). Yes, logs still spammed with snmpd messages

Is your VM an image of the real system? And the problems occur there too? If not, can you afford a real system reboot?

Not a real "ISO" image except for Debian 11... so, snmpd logs problem doesn't occur. Just trying to stop services and watch.
Can't afford a real system reboot before to check snapshot restore as it is a cloud VM. (a little chilly yes...)

Yes, you are right. Take a look at the systemd-journald manual though, search for "Size".
systemd-journald seems to be more resilient to this type of problem. Try starting it while keeping rsyslog stopped and see what happens.

Good explanation there... (in FR but without a doubt available in EN) :
https://www.digitalocean.com/community/ ... ux-recents

Thx to share Fabien!

[Aki]

Hello @486DX2,

Have you sorted it out ?

[486DX2]

Hello and sorry to be late...


Finally i stopped rsyslog:

Code: Select all

systemctl disable rsyslog
systemctl stop rsyslog

And overwrite (19 Gb files......) daemon.log.1 and syslog.1 :

Code: Select all

echo 1 > /tmp/daemon.log.1
	-rw-r--r--  1 root root   70 17 juil. 11:21 daemon.log.1
chmod 640 /tmp/daemon.log.1
	-rw-r-----  1 root root   70 17 juil. 11:21 daemon.log.1
chown root:adm /tmp/daemon.log.1
	-rw-r-----  1 root adm    70 17 juil. 11:21 daemon.log.1

Code: Select all

cp -p /tmp/daemon.log.1 /tmp/syslog.1

Code: Select all

cp /tmp/daemon.log.1 /var/log/daemon.log.1
cp /tmp/syslog.1 /var/log/syslog.1

Code: Select all

systemctl enable rsyslog.service
systemctl start rsyslog.service

I have changed /etc/logrotate.d/rsyslog to add :
size 10M
after
weekly

and force a logrotate:

Code: Select all

logrotate /etc/logrotate.d/rsyslog

This "puts out the fire but leaves the embers" but it could let me more time to try to investigate about these boring SNMP error messages...

Thanks to help & share!