[Software] Systemd / journalctl issue

[sioban]

I've discovered I have an issue with systemd on my Debian server. I've seen that some logging service don't want to start through systemd but the daemon itself start without issues:

Code: Select all

❯ systemctl start rsyslog.service
A dependency job for rsyslog.service failed. See 'journalctl -xe' for details.

If I start manually rsyslogd, it's working:

Code: Select all

❯ /usr/sbin/rsyslogd
❯ ps awx | grep rsyslog
  45995 ?        Ssl    0:00 /usr/sbin/rsyslogd

So I tried to understand what's wrong with systemd.

First, journalctl -xe shows only lines from 24 Oct, nothing recent.

I've tried timedatectl to ensure the date is correct but got this message

Code: Select all

❯ timedatectl
Failed to connect to bus: No such file or directory

I've searched a lot on Internet and ended up running systemctl --failed command and it shown many failed services:

Code: Select all

❯ systemctl --failed
  UNIT                            LOAD   ACTIVE SUB    DESCRIPTION
● dbus.socket                     loaded failed failed D-Bus System Message Bus Socket
● syslog.socket                   loaded failed failed Syslog Socket
● systemd-fsckd.socket            loaded failed failed fsck to fsckd communication Socket
● systemd-journald-audit.socket   loaded failed failed Journal Audit Socket
● systemd-journald-dev-log.socket loaded failed failed Journal Socket (/dev/log)
● systemd-journald.socket         loaded failed failed Journal Socket
● systemd-udevd-control.socket    loaded failed failed udev Control Socket
● systemd-udevd-kernel.socket     loaded failed failed udev Kernel Socket
● uuidd.socket                    loaded failed failed UUID daemon activation socket

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
9 loaded units listed.

I've re-installed systemd-sysv and rebooted, cleaned old journalctl entries but same issues.

Basically, I'm clueless here :/ If someone have an idea

[cds60601]

"Failed to connect to bus: No such file or directory"

Is dbus running?
From my Debian server on my Pi:

Code: Select all

chris@earth:~$ systemctl status dbus
● dbus.service - D-Bus System Message Bus
     Loaded: loaded (/lib/systemd/system/dbus.service; static)
     Active: active (running) since Fri 2022-11-11 04:44:45 EST; 1 months 14 days ago
TriggeredBy: ● dbus.socket
       Docs: man:dbus-daemon(1)
   Main PID: 449 (dbus-daemon)
      Tasks: 1 (limit: 9255)
     Memory: 1.8M
        CPU: 4.995s
     CGroup: /system.slice/dbus.service
             └─449 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only

[sioban]

hi, didn't see your comment

No DBus is not running.

Code: Select all

❯ systemctl status dbus
● dbus.service - D-Bus System Message Bus
     Loaded: loaded (/lib/systemd/system/dbus.service; static)
     Active: inactive (dead)
TriggeredBy: ● dbus.socket
       Docs: man:dbus-daemon(1)

[sioban]

here some more details

Code: Select all

❯ df /var/log /var/log/journal
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/md24       46096212 1620728  42101496   4% /var/log
/dev/md24       46096212 1620728  42101496   4% /var/log

Code: Select all

❯ systemctl status rsyslog.service
● rsyslog.service - System Logging Service
     Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
TriggeredBy: ● syslog.socket
       Docs: man:rsyslogd(8)
             man:rsyslog.conf(5)
             https://www.rsyslog.com/doc/

Code: Select all

❯ journalctl -u rsyslog.service | tail
-- Journal begins at Sun 2022-10-23 19:55:11 CEST, ends at Mon 2022-10-24 14:14:33 CEST. --
Oct 24 14:14:02 shax systemd[1]: Stopping System Logging Service...
Oct 24 14:14:02 shax systemd[1]: rsyslog.service: Succeeded.
Oct 24 14:14:02 shax systemd[1]: Stopped System Logging Service.
Oct 24 14:14:02 shax systemd[1]: rsyslog.service: Consumed 47.066s CPU time.

[sioban]

I'm also posting more information as I'm working on the issue with debian user mailing-list.
if it can give you some hints:

Code: Select all

❯ systemctl cat rsyslog.service
# /lib/systemd/system/rsyslog.service
[Unit]
Description=System Logging Service
Requires=syslog.socket
Documentation=man:rsyslogd(8)
Documentation=man:rsyslog.conf(5)
Documentation=https://www.rsyslog.com/doc/

[Service]
Type=notify
ExecStart=/usr/sbin/rsyslogd -n -iNONE
StandardOutput=null
Restart=on-failure

# Increase the default a bit in order to allow many simultaneous
# files to be monitored, we might need a lot of fds.
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target
Alias=syslog.service

Code: Select all

❯ systemctl status syslog.socket
● syslog.socket - Syslog Socket
     Loaded: loaded (/lib/systemd/system/syslog.socket; static)
     Active: failed (Result: resources)
   Triggers: ● rsyslog.service
       Docs: man:systemd.special(7)
             https://www.freedesktop.org/wiki/Software/systemd/syslog
     Listen: /run/systemd/journal/syslog (Datagram)

Code: Select all

❯ systemctl cat syslog.socket
# /lib/systemd/system/syslog.socket
#  SPDX-License-Identifier: LGPL-2.1-or-later
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Syslog Socket
Documentation=man:systemd.special(7)
Documentation=https://www.freedesktop.org/wiki/Software/systemd/syslog
DefaultDependencies=no
Before=sockets.target

# Don't allow logging until the very end
Conflicts=shutdown.target
Before=shutdown.target

# Don't try to activate syslog.service if sysinit.target has failed.
Conflicts=emergency.service
Before=emergency.service

[Socket]
ListenDatagram=/run/systemd/journal/syslog
SocketMode=0666
PassCredentials=yes
PassSecurity=yes
ReceiveBuffer=8M

# The default syslog implementation should make syslog.service a
# symlink to itself, so that this socket activates the right actual
# syslog service.
#
# Examples:
#
# /etc/systemd/system/syslog.service -> /lib/systemd/system/rsyslog.service
# /etc/systemd/system/syslog.service -> /lib/systemd/system/syslog-ng.service
#
# Best way to achieve that is by adding this to your unit file
# (i.e. to rsyslog.service or syslog-ng.service):
#
# [Install]
# Alias=syslog.service
#
# See https://www.freedesktop.org/wiki/Software/systemd/syslog for details.

Code: Select all

❯ systemctl start syslog.socket
Job failed. See "journalctl -xe" for details.

To be noted: I've started rsyslogd by hand using "/usr/sbin/rsyslogd -i /run/rsyslogd.pid" and there is nothing in the logs about the failure

[sioban]

Hi,

Here are some progress but the problem is still not solved.

/run/systemd/journal was missing.
So I've reinstalled some packages.

Code: Select all

❯ apt-get install --reinstall  systemd systemd-sysv libsystemd0 dbus-user-session
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 4 reinstalled, 0 to remove and 3 not upgraded.
Need to get 376 kB/5,090 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://ftp.fr.debian.org//debian bullseye/main amd64 libsystemd0 amd64 247.3-7+deb11u1 [376 kB]
Fetched 376 kB in 1s (645 kB/s)
(Reading database ... 204034 files and directories currently installed.)
Preparing to unpack .../libsystemd0_247.3-7+deb11u1_amd64.deb ...
Unpacking libsystemd0:amd64 (247.3-7+deb11u1) over (247.3-7+deb11u1) ...
Setting up libsystemd0:amd64 (247.3-7+deb11u1) ...
(Reading database ... 204034 files and directories currently installed.)
Preparing to unpack .../systemd_247.3-7+deb11u1_amd64.deb ...
Unpacking systemd (247.3-7+deb11u1) over (247.3-7+deb11u1) ...
Setting up systemd (247.3-7+deb11u1) ...
(Reading database ... 204034 files and directories currently installed.)
Preparing to unpack .../systemd-sysv_247.3-7+deb11u1_amd64.deb ...
Unpacking systemd-sysv (247.3-7+deb11u1) over (247.3-7+deb11u1) ...
Preparing to unpack .../dbus-user-session_1.12.24-0+deb11u1_amd64.deb ...
Unpacking dbus-user-session (1.12.24-0+deb11u1) over (1.12.24-0+deb11u1) ...
Setting up systemd-sysv (247.3-7+deb11u1) ...
Setting up dbus-user-session (1.12.24-0+deb11u1) ...
Processing triggers for libc-bin (2.31-13+deb11u5) ...
Processing triggers for man-db (2.9.4-2) ...
Processing triggers for dbus (1.12.24-0+deb11u1) ...
Scanning processes...
Scanning candidates...
Scanning processor microcode...
Scanning linux images...

Running kernel seems to be up-to-date.

The processor microcode seems to be up-to-date.

Restarting services...
 systemctl restart clamav-daemon.service ossec.service smartmontools.service squid.service ssh.service tor@default.service transmission-daemon.service
Service restarts being deferred:
 systemctl restart unattended-upgrades.service

No containers need to be restarted.

No user sessions are running outdated binaries.

It helped!
The folders have been recreated:

Code: Select all

❯ ls -la /run/systemd/journal/socket
srw-rw-rw- 1 root root 0 Feb  1 15:44 /run/systemd/journal/socket

❯ ls -lad /run/systemd/journal/
drwxr-xr-x 2 root root 120 Feb  1 15:44 /run/systemd/journal/

I was even able to start most of the failing process but two:

Code: Select all

❯ systemctl --failed
  UNIT                         LOAD   ACTIVE SUB    DESCRIPTION
● systemd-udevd-control.socket loaded failed failed udev Control Socket
● systemd-udevd-kernel.socket  loaded failed failed udev Kernel Socket

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
2 loaded units listed.

However journalctl still stucks on 24th Oct:

Code: Select all

❯ journalctl -b
-- Journal begins at Sun 2022-10-23 19:55:11 CEST, ends at Mon 2022-10-24 14:14:33 CEST. --
-- No entries --

I think it's linked with the fact that I have this entry in /var/log/journal (not /run/systemd/journal)

Code: Select all

❯ ls -la journal/
total 44
drwxr-sr-x+  3 root systemd-journal  4096 Aug 19  2021 .
drwxr-xr-x. 33 root root            12288 Feb  1 06:25 ..
drwxr-sr-x+  2 root systemd-journal 20480 Dec 24 11:35 ef72ee158c9d4ae68bd60ae8adbb20af

❯ ls -la journal/ef72ee158c9d4ae68bd60ae8adbb20af/
total 98356
drwxr-sr-x+ 2 root systemd-journal    20480 Dec 24 11:35 .
drwxr-sr-x+ 3 root systemd-journal     4096 Aug 19  2021 ..
-rw-r-----+ 1 root systemd-journal 75497472 Oct 24 14:14 system.journal
-rw-r-----+ 1 root systemd-journal  8388608 Oct 24 14:14 user-1000.journal
-rw-r-----+ 1 root systemd-journal  8388608 Oct 23 19:56 user-1014.journal
-rw-r-----+ 1 root systemd-journal  8388608 Oct 24 14:14 user-1015.journal

However, I have some files that are created in /run/systemd/journal

Code: Select all

❯ ls -la /run/systemd/journal/
total 0
drwxr-xr-x  2 root root 120 Feb  1 15:44 .
drwxr-xr-x 19 root root 460 Feb  1 15:43 ..
srw-rw-rw-  1 root root   0 Feb  1 15:44 dev-log
srw-rw-rw-  1 root root   0 Feb  1 15:44 socket
srw-rw-rw-  1 root root   0 Feb  1 15:44 stdout
srw-rw-rw-  1 root root   0 Feb  1 15:43 syslog