systemd using a lot of memory on server

Kernels & Hardware, configuring network, installing services

systemd using a lot of memory on server

Postby styxxx » 2018-07-12 14:46

Hi,

I'm running a small headless server (debian stable, 2.6.32-042stab129.1) and noticed that the process that uses by far the most memory is systemd with around 20% overall memory usage. I think that's pretty weird.

systemd-cgtop shows:
Code: Select all
Control Group                                                                                                                                                                                          Procs   %CPU   Memory  Input/s Output/s
/                                                                                                                                                                                                         26      -   746.8M        -        -
/init.scope                                                                                                                                                                                                1      -        -        -        -
/system.slice                                                                                                                                                                                             25      -        -        -        -
/system.slice/console-getty.service                                                                                                                                                                        1      -        -        -        -
/system.slice/cron.service                                                                                                                                                                                 1      -        -        -        -
/system.slice/dbus.service                                                                                                                                                                                 1      -        -        -        -
/system.slice/exim4.service                                                                                                                                                                                1      -        -        -        -
/system.slice/fail2ban.service                                                                                                                                                                             1      -        -        -        -
/system.slice/lighttpd.service                                                                                                                                                                             1      -        -        -        -
/system.slice/php7.0-fpm.service                                                                                                                                                                           3      -        -        -        -
/system.slice/prosody.service                                                                                                                                                                              1      -        -        -        -
/system.slice/rsyslog.service                                                                                                                                                                              1      -        -        -        -
/system.slice/ssh.service                                                                                                                                                                                  8      -        -        -        -
/system.slice/system-getty.slice                                                                                                                                                                           1      -        -        -        -
/system.slice/system-getty.slice/getty@tty2.service                                                                                                                                                        1      -        -        -        -
/system.slice/systemd-journald.service                                                                                                                                                                     1      -        -        -        -
/system.slice/systemd-logind.service                                                                                                                                                                       1      -        -        -        -
/system.slice/systemd-networkd.service                                                                                                                                                                     1      -        -        -        -
/system.slice/systemd-resolved.service                                                                                                                                                                     1      -        -        -        -
/system.slice/systemd-udevd.service                                                                                                                                                                        1      -        -        -        -


Installed version:
Code: Select all
# dpkg -l | grep systemd
ii  libpam-systemd:amd64          232-25+deb9u3                  amd64        system and service manager - PAM module
ii  libsystemd0:amd64             232-25+deb9u3                  amd64        systemd utility library
ii  python3-systemd               233-1                          amd64        Python 3 bindings for systemd
ii  systemd                       232-25+deb9u3                  amd64        system and service manager
ii  systemd-sysv                  232-25+deb9u3                  amd64        system and service manager - SysV links


journalctl doesn't show anything that might explain it. Although there are some strange messages every minute, it's always the same:
Code: Select all
Jul 12 17:23:07 servername dbus[15418]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out
Jul 12 17:23:07 servername systemd-logind[15447]: Failed to enable subscription: Failed to activate service 'org.freedesktop.systemd1': timed out
Jul 12 17:23:07 servername systemd-logind[15447]: Failed to fully start up daemon: Connection timed out
Jul 12 17:23:07 servername systemd[1]: systemd-logind.service: Main process exited, code=exited, status=1/FAILURE
Jul 12 17:23:07 servername systemd[1]: Failed to start Login Service.
Jul 12 17:23:07 servername systemd[1]: systemd-logind.service: Unit entered failed state.
Jul 12 17:23:07 servername systemd[1]: systemd-logind.service: Failed with result 'exit-code'.
Jul 12 17:23:07 servername systemd[1]: systemd-logind.service: Service has no hold-off time, scheduling restart.
Jul 12 17:23:07 servername systemd[1]: Stopped Login Service.
Jul 12 17:23:07 servername systemd[1]: systemd-logind.service: Failed to set invocation ID on control group /system.slice/systemd-logind.service, ignoring: Operation not permitted
Jul 12 17:23:07 servername systemd[1]: Starting Login Service...


Also "journalctl -xe" after manually executing "systemctl restart systemd-logind" (that fails) shows:
Code: Select all
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit systemd-logind.service has failed.
--
-- The result is failed.
Jul 12 17:27:17 servername systemd[1]: systemd-logind.service: Unit entered failed state.
Jul 12 17:27:17 servername systemd[1]: systemd-logind.service: Failed with result 'exit-code'.
Jul 12 17:27:17 servername systemd[1]: systemd-logind.service: Service has no hold-off time, scheduling restart.
Jul 12 17:27:17 servername systemd[1]: Stopped Login Service.
-- Subject: Unit systemd-logind.service has finished shutting down
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit systemd-logind.service has finished shutting down.
Jul 12 17:27:17 servername systemd[1]: systemd-logind.service: Failed to set invocation ID on control group /system.slice/systemd-logind.service, ignoring: Operation not permitted
Jul 12 17:27:17 servername systemd[1]: Starting Login Service...
-- Subject: Unit systemd-logind.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit systemd-logind.service has begun starting up.
Jul 12 17:27:36 servername systemd[1]: Stopped Login Service.
-- Subject: Unit systemd-logind.service has finished shutting down
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit systemd-logind.service has finished shutting down.
Jul 12 17:27:36 servername systemd[1]: systemd-logind.service: Failed to set invocation ID on control group /system.slice/systemd-logind.service, ignoring: Operation not permitted
Jul 12 17:27:36 servername systemd[1]: Starting Login Service...
-- Subject: Unit systemd-logind.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit systemd-logind.service has begun starting up.
Jul 12 17:27:42 servername dbus[15418]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out
Jul 12 17:27:42 servername systemd-logind[15480]: Failed to enable subscription: Failed to activate service 'org.freedesktop.systemd1': timed out
Jul 12 17:27:42 servername systemd-logind[15480]: Failed to fully start up daemon: Connection timed out
Jul 12 17:27:42 servername systemd[1]: systemd-logind.service: Main process exited, code=exited, status=1/FAILURE
Jul 12 17:27:42 servername systemd[1]: Failed to start Login Service.
-- Subject: Unit systemd-logind.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit systemd-logind.service has failed.
--
-- The result is failed.
Jul 12 17:27:42 servername systemd[1]: systemd-logind.service: Unit entered failed state.
Jul 12 17:27:42 servername systemd[1]: systemd-logind.service: Failed with result 'exit-code'.
Jul 12 17:27:42 servername systemd[1]: systemd-logind.service: Service has no hold-off time, scheduling restart.
Jul 12 17:27:42 servername systemd[1]: Stopped Login Service.
-- Subject: Unit systemd-logind.service has finished shutting down
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit systemd-logind.service has finished shutting down.
Jul 12 17:27:42 servername systemd[1]: systemd-logind.service: Failed to set invocation ID on control group /system.slice/systemd-logind.service, ignoring: Operation not permitted
Jul 12 17:27:42 servername systemd[1]: Starting Login Service...
-- Subject: Unit systemd-logind.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit systemd-logind.service has begun starting up.


So while this is annyoing and spamming my system log I don't think it's the cause of the huge memory usage.

Anyone any idea how to fix this and find the cause of systmd's possible memory leak?
styxxx
 
Posts: 18
Joined: 2013-06-03 20:24

Re: systemd using a lot of memory on server

Postby bw123 » 2018-07-12 17:05

debian stable, 2.6.32-042stab129.1


hmmm, debian stable kernel 2.6 that doesn't sound right... I get this on stretch:
Code: Select all
uname -a
Linux hostname 4.9.0-6-amd64 #1 SMP Debian 4.9.88-1+deb9u1 (2018-05-07) x86_64 GNU/Linux


I'm not real good with systemd, but from what you posted I don't see anything about a memory leak or anywhere that says systemd is using a lot of memory, or 20% of your memory?

I don't know about systemd-cgtop but I will read up on it thanks. Do you see how much systemd is using of my memory here?

my output:
Code: Select all
Control Group                            Tasks   %CPU   Memory  Input/s Output/s
/                                            -    4.8   376.0M        -        -
/init.scope                                  1      -        -        -        -
/system.slice                               22      -        -        -        -
/system.slice/acpid.service                  1      -        -        -        -
/system.slice/autofs.service                 4      -        -        -        -
/system.slice/cron.service                   1      -        -        -        -
/system.slice/dbus.service                   1      -        -        -        -
/system.slice/exim4.service                  1      -        -        -        -
/system.slice/gpm.service                    1      -        -        -        -
/system.slice/ifup@wlan0.service             2      -        -        -        -
/system.slice/rsyslog.service                4      -        -        -        -
/system.slice/smartd.service                 1      -        -        -        -
/system.slice/systemd-journald.service       1      -        -        -        -
/system.slice/systemd-logind.service         1      -        -        -        -
/system.slic…systemd-timesyncd.service       2      -        -        -        -
/system.slice/systemd-udevd.service          1      -        -        -        -
/system.slice/vnstat.service                 1      -        -        -        -
/user.slice                                 38      -        -        -        -
/user.slice/user-1000.slice                 38      -        -        -        -
/user.slice/…000.slice/session-1.scope      36      -        -        -        -
$ man systemd-cgtop

User avatar
bw123
 
Posts: 3580
Joined: 2011-05-09 06:02
Location: TN_USA

Re: systemd using a lot of memory on server

Postby GarryRicketson » 2018-07-12 18:50

by bw123 »
hmmm, debian stable kernel 2.6 that doesn't sound right


No it does not sound right at all, 2.6 was for Debian squeeze,
kernel : linux 2.6.32

Something just doesn't sound right, to me either. Maybe the OP can clarify,
the rest of it's details may not be accurate either, this just does not sound right.
User avatar
GarryRicketson
 
Posts: 5193
Joined: 2015-01-20 22:16
Location: Durango, Mexico

Re: systemd using a lot of memory on server

Postby styxxx » 2018-07-13 02:42

Hm, thank you! That's indeed weird. :oops:
It's an OpenVZ vserver and I installed the OS via the solusVM web panel a while ago. The panel currently says "Debian 8.0 64 Bit minimal". So it would be oldstable not stable (my mistake: 8 was stable when I installed it). But even then the kernel would be to old - but is afaik provided by the openvz host, so I can't upgrade it. The default repository entries are for "jessie" and "stable" :roll: and all available upgrades (just some minor updates like php and perl, no critical packages) got installed today.


So something's wrong there. But could this be the reason for systemd using so much memory? It seems to grow slowly (unfortunately I didn't record the exact numbers) that's why I came up with the word "leak". I mean, is there a scenario where systemd would use hundreds of megabytes on purpose?
I also noticed that a services failed for unknown reasons (a backup script). It seemed to be frozen during the last execution. But there were no (error) messages in the systemd journal except one about me killing it. But I don't think that messes with the memory and killing it didn't change anything.

The system was running fine for months without someone touching it (at least not the container). If it was because of an old kernel I'd expect some error messages mentioning that and not extensive memory usage :wink: But apart from those entries about org.freedesktop.systemd1 there are none and everything else just seems to work. :?

Code: Select all
# dpkg -l | egrep -e "linux|header"
ii  libselinux1:amd64             2.6-3+b3                       amd64        SELinux runtime shared libraries
ii  libselinux1-dev:amd64         2.6-3+b3                       amd64        SELinux development headers
ii  linux-base                    4.5                            all          Linux image base package
ii  linux-libc-dev:amd64          4.9.88-1+deb9u1                amd64        Linux support headers for userspace development
ii  util-linux                    2.29.2-1+deb9u1                amd64        miscellaneous system utilities

Code: Select all
# uname -srvmpio
Linux 2.6.32-042stab129.1 #1 SMP Thu May 10 13:24:14 MSK 2018 x86_64 unknown unknown GNU/Linux


Code: Select all
# cat /etc/apt/sources.list.d/*
deb http://ftp.debian.org/debian jessie main contrib non-free
deb http://ftp.debian.org/debian jessie-updates main contrib non-free
deb http://security.debian.org jessie/updates main contrib non-free

deb http://ftp.debian.org/debian stable main contrib non-free
deb http://ftp.debian.org/debian stable-updates main contrib non-free
deb http://security.debian.org stable/updates main contrib non-free


Code: Select all
# apt-get dist-upgrade
Paketlisten werden gelesen... Fertig
Abhängigkeitsbaum wird aufgebaut.       
Statusinformationen werden eingelesen.... Fertig
Paketaktualisierung (Upgrade) wird berechnet... Fertig
0 aktualisiert, 0 neu installiert, 0 zu entfernen und 0 nicht aktualisiert.
styxxx
 
Posts: 18
Joined: 2013-06-03 20:24

Re: systemd using a lot of memory on server

Postby Head_on_a_Stick » 2018-07-13 04:11

styxxx wrote:could this be the reason for systemd using so much memory? It seems to grow slowly

The only memory figure you posted so far is the general readout from systemd-cgtop — this is *not* in any way an accurate measure of memory usage, my Debian system displays a completely different value if the `free` command is used.

Can we please see
Code: Select all
free -h # at boot and after this increase you claim to have observed
cat /proc/version
apt policy linux-image-amd64

I like to use ps_mem to analyse memory usage, it needs to be run as root but it's just a simple python script.
dbruce wrote:Ubuntu forums try to be like a coffee shop in Seattle. Debian forums strive for the charm and ambience of a skinhead bar in Bacau. We intend to keep it that way.
User avatar
Head_on_a_Stick
 
Posts: 8322
Joined: 2014-06-01 17:46
Location: /dev/chair


Return to System configuration

Who is online

Users browsing this forum: No registered users and 7 guests

fashionable