Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

systemd using a lot of memory on server

Linux Kernel, Network, and Services configuration.
Post Reply
Message
Author
styxxx
Posts: 19
Joined: 2013-06-03 20:24

systemd using a lot of memory on server

#1 Post by styxxx »

Hi,

I'm running a small headless server (debian stable, 2.6.32-042stab129.1) and noticed that the process that uses by far the most memory is systemd with around 20% overall memory usage. I think that's pretty weird.

systemd-cgtop shows:

Code: Select all

Control Group                                                                                                                                                                                          Procs   %CPU   Memory  Input/s Output/s
/                                                                                                                                                                                                         26      -   746.8M        -        -
/init.scope                                                                                                                                                                                                1      -        -        -        -
/system.slice                                                                                                                                                                                             25      -        -        -        -
/system.slice/console-getty.service                                                                                                                                                                        1      -        -        -        -
/system.slice/cron.service                                                                                                                                                                                 1      -        -        -        -
/system.slice/dbus.service                                                                                                                                                                                 1      -        -        -        -
/system.slice/exim4.service                                                                                                                                                                                1      -        -        -        -
/system.slice/fail2ban.service                                                                                                                                                                             1      -        -        -        -
/system.slice/lighttpd.service                                                                                                                                                                             1      -        -        -        -
/system.slice/php7.0-fpm.service                                                                                                                                                                           3      -        -        -        -
/system.slice/prosody.service                                                                                                                                                                              1      -        -        -        -
/system.slice/rsyslog.service                                                                                                                                                                              1      -        -        -        -
/system.slice/ssh.service                                                                                                                                                                                  8      -        -        -        -
/system.slice/system-getty.slice                                                                                                                                                                           1      -        -        -        -
/system.slice/system-getty.slice/getty@tty2.service                                                                                                                                                        1      -        -        -        -
/system.slice/systemd-journald.service                                                                                                                                                                     1      -        -        -        -
/system.slice/systemd-logind.service                                                                                                                                                                       1      -        -        -        -
/system.slice/systemd-networkd.service                                                                                                                                                                     1      -        -        -        -
/system.slice/systemd-resolved.service                                                                                                                                                                     1      -        -        -        -
/system.slice/systemd-udevd.service                                                                                                                                                                        1      -        -        -        -
Installed version:

Code: Select all

# dpkg -l | grep systemd
ii  libpam-systemd:amd64          232-25+deb9u3                  amd64        system and service manager - PAM module
ii  libsystemd0:amd64             232-25+deb9u3                  amd64        systemd utility library
ii  python3-systemd               233-1                          amd64        Python 3 bindings for systemd
ii  systemd                       232-25+deb9u3                  amd64        system and service manager
ii  systemd-sysv                  232-25+deb9u3                  amd64        system and service manager - SysV links
journalctl doesn't show anything that might explain it. Although there are some strange messages every minute, it's always the same:

Code: Select all

Jul 12 17:23:07 servername dbus[15418]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out
Jul 12 17:23:07 servername systemd-logind[15447]: Failed to enable subscription: Failed to activate service 'org.freedesktop.systemd1': timed out
Jul 12 17:23:07 servername systemd-logind[15447]: Failed to fully start up daemon: Connection timed out
Jul 12 17:23:07 servername systemd[1]: systemd-logind.service: Main process exited, code=exited, status=1/FAILURE
Jul 12 17:23:07 servername systemd[1]: Failed to start Login Service.
Jul 12 17:23:07 servername systemd[1]: systemd-logind.service: Unit entered failed state.
Jul 12 17:23:07 servername systemd[1]: systemd-logind.service: Failed with result 'exit-code'.
Jul 12 17:23:07 servername systemd[1]: systemd-logind.service: Service has no hold-off time, scheduling restart.
Jul 12 17:23:07 servername systemd[1]: Stopped Login Service.
Jul 12 17:23:07 servername systemd[1]: systemd-logind.service: Failed to set invocation ID on control group /system.slice/systemd-logind.service, ignoring: Operation not permitted
Jul 12 17:23:07 servername systemd[1]: Starting Login Service...
Also "journalctl -xe" after manually executing "systemctl restart systemd-logind" (that fails) shows:

Code: Select all

-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit systemd-logind.service has failed.
-- 
-- The result is failed.
Jul 12 17:27:17 servername systemd[1]: systemd-logind.service: Unit entered failed state.
Jul 12 17:27:17 servername systemd[1]: systemd-logind.service: Failed with result 'exit-code'.
Jul 12 17:27:17 servername systemd[1]: systemd-logind.service: Service has no hold-off time, scheduling restart.
Jul 12 17:27:17 servername systemd[1]: Stopped Login Service.
-- Subject: Unit systemd-logind.service has finished shutting down
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit systemd-logind.service has finished shutting down.
Jul 12 17:27:17 servername systemd[1]: systemd-logind.service: Failed to set invocation ID on control group /system.slice/systemd-logind.service, ignoring: Operation not permitted
Jul 12 17:27:17 servername systemd[1]: Starting Login Service...
-- Subject: Unit systemd-logind.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit systemd-logind.service has begun starting up.
Jul 12 17:27:36 servername systemd[1]: Stopped Login Service.
-- Subject: Unit systemd-logind.service has finished shutting down
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit systemd-logind.service has finished shutting down.
Jul 12 17:27:36 servername systemd[1]: systemd-logind.service: Failed to set invocation ID on control group /system.slice/systemd-logind.service, ignoring: Operation not permitted
Jul 12 17:27:36 servername systemd[1]: Starting Login Service...
-- Subject: Unit systemd-logind.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit systemd-logind.service has begun starting up.
Jul 12 17:27:42 servername dbus[15418]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out
Jul 12 17:27:42 servername systemd-logind[15480]: Failed to enable subscription: Failed to activate service 'org.freedesktop.systemd1': timed out
Jul 12 17:27:42 servername systemd-logind[15480]: Failed to fully start up daemon: Connection timed out
Jul 12 17:27:42 servername systemd[1]: systemd-logind.service: Main process exited, code=exited, status=1/FAILURE
Jul 12 17:27:42 servername systemd[1]: Failed to start Login Service.
-- Subject: Unit systemd-logind.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit systemd-logind.service has failed.
-- 
-- The result is failed.
Jul 12 17:27:42 servername systemd[1]: systemd-logind.service: Unit entered failed state.
Jul 12 17:27:42 servername systemd[1]: systemd-logind.service: Failed with result 'exit-code'.
Jul 12 17:27:42 servername systemd[1]: systemd-logind.service: Service has no hold-off time, scheduling restart.
Jul 12 17:27:42 servername systemd[1]: Stopped Login Service.
-- Subject: Unit systemd-logind.service has finished shutting down
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit systemd-logind.service has finished shutting down.
Jul 12 17:27:42 servername systemd[1]: systemd-logind.service: Failed to set invocation ID on control group /system.slice/systemd-logind.service, ignoring: Operation not permitted
Jul 12 17:27:42 servername systemd[1]: Starting Login Service...
-- Subject: Unit systemd-logind.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit systemd-logind.service has begun starting up.
So while this is annyoing and spamming my system log I don't think it's the cause of the huge memory usage.

Anyone any idea how to fix this and find the cause of systmd's possible memory leak?

User avatar
bw123
Posts: 4015
Joined: 2011-05-09 06:02
Has thanked: 1 time
Been thanked: 28 times

Re: systemd using a lot of memory on server

#2 Post by bw123 »

debian stable, 2.6.32-042stab129.1
hmmm, debian stable kernel 2.6 that doesn't sound right... I get this on stretch:

Code: Select all

uname -a
Linux hostname 4.9.0-6-amd64 #1 SMP Debian 4.9.88-1+deb9u1 (2018-05-07) x86_64 GNU/Linux
I'm not real good with systemd, but from what you posted I don't see anything about a memory leak or anywhere that says systemd is using a lot of memory, or 20% of your memory?

I don't know about systemd-cgtop but I will read up on it thanks. Do you see how much systemd is using of my memory here?

my output:

Code: Select all

Control Group                            Tasks   %CPU   Memory  Input/s Output/s
/                                            -    4.8   376.0M        -        -
/init.scope                                  1      -        -        -        -
/system.slice                               22      -        -        -        -
/system.slice/acpid.service                  1      -        -        -        -
/system.slice/autofs.service                 4      -        -        -        -
/system.slice/cron.service                   1      -        -        -        -
/system.slice/dbus.service                   1      -        -        -        -
/system.slice/exim4.service                  1      -        -        -        -
/system.slice/gpm.service                    1      -        -        -        -
/system.slice/ifup@wlan0.service             2      -        -        -        -
/system.slice/rsyslog.service                4      -        -        -        -
/system.slice/smartd.service                 1      -        -        -        -
/system.slice/systemd-journald.service       1      -        -        -        -
/system.slice/systemd-logind.service         1      -        -        -        -
/system.slic…systemd-timesyncd.service       2      -        -        -        -
/system.slice/systemd-udevd.service          1      -        -        -        -
/system.slice/vnstat.service                 1      -        -        -        -
/user.slice                                 38      -        -        -        -
/user.slice/user-1000.slice                 38      -        -        -        -
/user.slice/…000.slice/session-1.scope      36      -        -        -        -
$ man systemd-cgtop

resigned by AI ChatGPT

User avatar
GarryRicketson
Posts: 5644
Joined: 2015-01-20 22:16
Location: Durango, Mexico

Re: systemd using a lot of memory on server

#3 Post by GarryRicketson »

by bw123 »
hmmm, debian stable kernel 2.6 that doesn't sound right
No it does not sound right at all, 2.6 was for Debian squeeze,
kernel : linux 2.6.32

Something just doesn't sound right, to me either. Maybe the OP can clarify,
the rest of it's details may not be accurate either, this just does not sound right.

styxxx
Posts: 19
Joined: 2013-06-03 20:24

Re: systemd using a lot of memory on server

#4 Post by styxxx »

Hm, thank you! That's indeed weird. :oops:
It's an OpenVZ vserver and I installed the OS via the solusVM web panel a while ago. The panel currently says "Debian 8.0 64 Bit minimal". So it would be oldstable not stable (my mistake: 8 was stable when I installed it). But even then the kernel would be to old - but is afaik provided by the openvz host, so I can't upgrade it. The default repository entries are for "jessie" and "stable" :roll: and all available upgrades (just some minor updates like php and perl, no critical packages) got installed today.


So something's wrong there. But could this be the reason for systemd using so much memory? It seems to grow slowly (unfortunately I didn't record the exact numbers) that's why I came up with the word "leak". I mean, is there a scenario where systemd would use hundreds of megabytes on purpose?
I also noticed that a services failed for unknown reasons (a backup script). It seemed to be frozen during the last execution. But there were no (error) messages in the systemd journal except one about me killing it. But I don't think that messes with the memory and killing it didn't change anything.

The system was running fine for months without someone touching it (at least not the container). If it was because of an old kernel I'd expect some error messages mentioning that and not extensive memory usage :wink: But apart from those entries about org.freedesktop.systemd1 there are none and everything else just seems to work. :?

Code: Select all

# dpkg -l | egrep -e "linux|header"
ii  libselinux1:amd64             2.6-3+b3                       amd64        SELinux runtime shared libraries
ii  libselinux1-dev:amd64         2.6-3+b3                       amd64        SELinux development headers
ii  linux-base                    4.5                            all          Linux image base package
ii  linux-libc-dev:amd64          4.9.88-1+deb9u1                amd64        Linux support headers for userspace development
ii  util-linux                    2.29.2-1+deb9u1                amd64        miscellaneous system utilities

Code: Select all

# uname -srvmpio
Linux 2.6.32-042stab129.1 #1 SMP Thu May 10 13:24:14 MSK 2018 x86_64 unknown unknown GNU/Linux

Code: Select all

# cat /etc/apt/sources.list.d/* 
deb http://ftp.debian.org/debian jessie main contrib non-free
deb http://ftp.debian.org/debian jessie-updates main contrib non-free
deb http://security.debian.org jessie/updates main contrib non-free

deb http://ftp.debian.org/debian stable main contrib non-free
deb http://ftp.debian.org/debian stable-updates main contrib non-free
deb http://security.debian.org stable/updates main contrib non-free

Code: Select all

# apt-get dist-upgrade
Paketlisten werden gelesen... Fertig
Abhängigkeitsbaum wird aufgebaut.       
Statusinformationen werden eingelesen.... Fertig
Paketaktualisierung (Upgrade) wird berechnet... Fertig
0 aktualisiert, 0 neu installiert, 0 zu entfernen und 0 nicht aktualisiert.

User avatar
Head_on_a_Stick
Posts: 14114
Joined: 2014-06-01 17:46
Location: London, England
Has thanked: 81 times
Been thanked: 132 times

Re: systemd using a lot of memory on server

#5 Post by Head_on_a_Stick »

styxxx wrote:could this be the reason for systemd using so much memory? It seems to grow slowly
The only memory figure you posted so far is the general readout from systemd-cgtop — this is *not* in any way an accurate measure of memory usage, my Debian system displays a completely different value if the `free` command is used.

Can we please see

Code: Select all

free -h # at boot and after this increase you claim to have observed
cat /proc/version
apt policy linux-image-amd64
I like to use ps_mem to analyse memory usage, it needs to be run as root but it's just a simple python script.
deadbang

Post Reply