Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

[Software] Systemd / journalctl issue

If none of the specific sub-forums seem right for your thread, ask here.
Post Reply
Message
Author
sioban
Posts: 20
Joined: 2015-04-29 19:48

[Software] Systemd / journalctl issue

#1 Post by sioban »

I've discovered I have an issue with systemd on my Debian server. I've seen that some logging service don't want to start through systemd but the daemon itself start without issues:

Code: Select all

❯ systemctl start rsyslog.service
A dependency job for rsyslog.service failed. See 'journalctl -xe' for details.
If I start manually rsyslogd, it's working:

Code: Select all

❯ /usr/sbin/rsyslogd
❯ ps awx | grep rsyslog
  45995 ?        Ssl    0:00 /usr/sbin/rsyslogd
So I tried to understand what's wrong with systemd.

First, journalctl -xe shows only lines from 24 Oct, nothing recent.

I've tried timedatectl to ensure the date is correct but got this message

Code: Select all

❯ timedatectl
Failed to connect to bus: No such file or directory
I've searched a lot on Internet and ended up running systemctl --failed command and it shown many failed services:

Code: Select all

❯ systemctl --failed
  UNIT                            LOAD   ACTIVE SUB    DESCRIPTION
● dbus.socket                     loaded failed failed D-Bus System Message Bus Socket
● syslog.socket                   loaded failed failed Syslog Socket
● systemd-fsckd.socket            loaded failed failed fsck to fsckd communication Socket
● systemd-journald-audit.socket   loaded failed failed Journal Audit Socket
● systemd-journald-dev-log.socket loaded failed failed Journal Socket (/dev/log)
● systemd-journald.socket         loaded failed failed Journal Socket
● systemd-udevd-control.socket    loaded failed failed udev Control Socket
● systemd-udevd-kernel.socket     loaded failed failed udev Kernel Socket
● uuidd.socket                    loaded failed failed UUID daemon activation socket

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
9 loaded units listed.
I've re-installed systemd-sysv and rebooted, cleaned old journalctl entries but same issues.

Basically, I'm clueless here :/ If someone have an idea :D

User avatar
cds60601
df -h | participant
df -h | participant
Posts: 698
Joined: 2017-11-25 05:58
Location: Florida
Has thanked: 129 times
Been thanked: 59 times

Re: [Software] Systemd / journalctl issue

#2 Post by cds60601 »

"Failed to connect to bus: No such file or directory"

Is dbus running?
From my Debian server on my Pi:

Code: Select all

chris@earth:~$ systemctl status dbus
● dbus.service - D-Bus System Message Bus
     Loaded: loaded (/lib/systemd/system/dbus.service; static)
     Active: active (running) since Fri 2022-11-11 04:44:45 EST; 1 months 14 days ago
TriggeredBy: ● dbus.socket
       Docs: man:dbus-daemon(1)
   Main PID: 449 (dbus-daemon)
      Tasks: 1 (limit: 9255)
     Memory: 1.8M
        CPU: 4.995s
     CGroup: /system.slice/dbus.service
             └─449 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
Supercalifragilisticexpialidocious

sioban
Posts: 20
Joined: 2015-04-29 19:48

Re: [Software] Systemd / journalctl issue

#3 Post by sioban »

hi, didn't see your comment :(

No DBus is not running.

Code: Select all

❯ systemctl status dbus
● dbus.service - D-Bus System Message Bus
     Loaded: loaded (/lib/systemd/system/dbus.service; static)
     Active: inactive (dead)
TriggeredBy: ● dbus.socket
       Docs: man:dbus-daemon(1)

sioban
Posts: 20
Joined: 2015-04-29 19:48

Re: [Software] Systemd / journalctl issue

#4 Post by sioban »

here some more details

Code: Select all

❯ df /var/log /var/log/journal
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/md24       46096212 1620728  42101496   4% /var/log
/dev/md24       46096212 1620728  42101496   4% /var/log

Code: Select all

❯ systemctl status rsyslog.service
● rsyslog.service - System Logging Service
     Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
TriggeredBy: ● syslog.socket
       Docs: man:rsyslogd(8)
             man:rsyslog.conf(5)
             https://www.rsyslog.com/doc/

Code: Select all

❯ journalctl -u rsyslog.service | tail
-- Journal begins at Sun 2022-10-23 19:55:11 CEST, ends at Mon 2022-10-24 14:14:33 CEST. --
Oct 24 14:14:02 shax systemd[1]: Stopping System Logging Service...
Oct 24 14:14:02 shax systemd[1]: rsyslog.service: Succeeded.
Oct 24 14:14:02 shax systemd[1]: Stopped System Logging Service.
Oct 24 14:14:02 shax systemd[1]: rsyslog.service: Consumed 47.066s CPU time.

sioban
Posts: 20
Joined: 2015-04-29 19:48

Re: [Software] Systemd / journalctl issue

#5 Post by sioban »

I'm also posting more information as I'm working on the issue with debian user mailing-list.
if it can give you some hints:

Code: Select all

❯ systemctl cat rsyslog.service
# /lib/systemd/system/rsyslog.service
[Unit]
Description=System Logging Service
Requires=syslog.socket
Documentation=man:rsyslogd(8)
Documentation=man:rsyslog.conf(5)
Documentation=https://www.rsyslog.com/doc/

[Service]
Type=notify
ExecStart=/usr/sbin/rsyslogd -n -iNONE
StandardOutput=null
Restart=on-failure

# Increase the default a bit in order to allow many simultaneous
# files to be monitored, we might need a lot of fds.
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target
Alias=syslog.service

Code: Select all

❯ systemctl status syslog.socket
● syslog.socket - Syslog Socket
     Loaded: loaded (/lib/systemd/system/syslog.socket; static)
     Active: failed (Result: resources)
   Triggers: ● rsyslog.service
       Docs: man:systemd.special(7)
             https://www.freedesktop.org/wiki/Software/systemd/syslog
     Listen: /run/systemd/journal/syslog (Datagram)

Code: Select all

❯ systemctl cat syslog.socket
# /lib/systemd/system/syslog.socket
#  SPDX-License-Identifier: LGPL-2.1-or-later
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Syslog Socket
Documentation=man:systemd.special(7)
Documentation=https://www.freedesktop.org/wiki/Software/systemd/syslog
DefaultDependencies=no
Before=sockets.target

# Don't allow logging until the very end
Conflicts=shutdown.target
Before=shutdown.target

# Don't try to activate syslog.service if sysinit.target has failed.
Conflicts=emergency.service
Before=emergency.service

[Socket]
ListenDatagram=/run/systemd/journal/syslog
SocketMode=0666
PassCredentials=yes
PassSecurity=yes
ReceiveBuffer=8M

# The default syslog implementation should make syslog.service a
# symlink to itself, so that this socket activates the right actual
# syslog service.
#
# Examples:
#
# /etc/systemd/system/syslog.service -> /lib/systemd/system/rsyslog.service
# /etc/systemd/system/syslog.service -> /lib/systemd/system/syslog-ng.service
#
# Best way to achieve that is by adding this to your unit file
# (i.e. to rsyslog.service or syslog-ng.service):
#
# [Install]
# Alias=syslog.service
#
# See https://www.freedesktop.org/wiki/Software/systemd/syslog for details.

Code: Select all

❯ systemctl start syslog.socket
Job failed. See "journalctl -xe" for details.
To be noted: I've started rsyslogd by hand using "/usr/sbin/rsyslogd -i /run/rsyslogd.pid" and there is nothing in the logs about the failure

sioban
Posts: 20
Joined: 2015-04-29 19:48

Re: [Software] Systemd / journalctl issue

#6 Post by sioban »

Hi,

Here are some progress but the problem is still not solved.

/run/systemd/journal was missing.
So I've reinstalled some packages.

Code: Select all

❯ apt-get install --reinstall  systemd systemd-sysv libsystemd0 dbus-user-session
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 4 reinstalled, 0 to remove and 3 not upgraded.
Need to get 376 kB/5,090 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://ftp.fr.debian.org//debian bullseye/main amd64 libsystemd0 amd64 247.3-7+deb11u1 [376 kB]
Fetched 376 kB in 1s (645 kB/s)
(Reading database ... 204034 files and directories currently installed.)
Preparing to unpack .../libsystemd0_247.3-7+deb11u1_amd64.deb ...
Unpacking libsystemd0:amd64 (247.3-7+deb11u1) over (247.3-7+deb11u1) ...
Setting up libsystemd0:amd64 (247.3-7+deb11u1) ...
(Reading database ... 204034 files and directories currently installed.)
Preparing to unpack .../systemd_247.3-7+deb11u1_amd64.deb ...
Unpacking systemd (247.3-7+deb11u1) over (247.3-7+deb11u1) ...
Setting up systemd (247.3-7+deb11u1) ...
(Reading database ... 204034 files and directories currently installed.)
Preparing to unpack .../systemd-sysv_247.3-7+deb11u1_amd64.deb ...
Unpacking systemd-sysv (247.3-7+deb11u1) over (247.3-7+deb11u1) ...
Preparing to unpack .../dbus-user-session_1.12.24-0+deb11u1_amd64.deb ...
Unpacking dbus-user-session (1.12.24-0+deb11u1) over (1.12.24-0+deb11u1) ...
Setting up systemd-sysv (247.3-7+deb11u1) ...
Setting up dbus-user-session (1.12.24-0+deb11u1) ...
Processing triggers for libc-bin (2.31-13+deb11u5) ...
Processing triggers for man-db (2.9.4-2) ...
Processing triggers for dbus (1.12.24-0+deb11u1) ...
Scanning processes...
Scanning candidates...
Scanning processor microcode...
Scanning linux images...

Running kernel seems to be up-to-date.

The processor microcode seems to be up-to-date.

Restarting services...
 systemctl restart clamav-daemon.service ossec.service smartmontools.service squid.service ssh.service tor@default.service transmission-daemon.service
Service restarts being deferred:
 systemctl restart unattended-upgrades.service

No containers need to be restarted.

No user sessions are running outdated binaries.
It helped!
The folders have been recreated:

Code: Select all

❯ ls -la /run/systemd/journal/socket
srw-rw-rw- 1 root root 0 Feb  1 15:44 /run/systemd/journal/socket

❯ ls -lad /run/systemd/journal/
drwxr-xr-x 2 root root 120 Feb  1 15:44 /run/systemd/journal/
I was even able to start most of the failing process but two:

Code: Select all

❯ systemctl --failed
  UNIT                         LOAD   ACTIVE SUB    DESCRIPTION
● systemd-udevd-control.socket loaded failed failed udev Control Socket
● systemd-udevd-kernel.socket  loaded failed failed udev Kernel Socket

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
2 loaded units listed.
However journalctl still stucks on 24th Oct:

Code: Select all

❯ journalctl -b
-- Journal begins at Sun 2022-10-23 19:55:11 CEST, ends at Mon 2022-10-24 14:14:33 CEST. --
-- No entries --
I think it's linked with the fact that I have this entry in /var/log/journal (not /run/systemd/journal)

Code: Select all

❯ ls -la journal/
total 44
drwxr-sr-x+  3 root systemd-journal  4096 Aug 19  2021 .
drwxr-xr-x. 33 root root            12288 Feb  1 06:25 ..
drwxr-sr-x+  2 root systemd-journal 20480 Dec 24 11:35 ef72ee158c9d4ae68bd60ae8adbb20af

❯ ls -la journal/ef72ee158c9d4ae68bd60ae8adbb20af/
total 98356
drwxr-sr-x+ 2 root systemd-journal    20480 Dec 24 11:35 .
drwxr-sr-x+ 3 root systemd-journal     4096 Aug 19  2021 ..
-rw-r-----+ 1 root systemd-journal 75497472 Oct 24 14:14 system.journal
-rw-r-----+ 1 root systemd-journal  8388608 Oct 24 14:14 user-1000.journal
-rw-r-----+ 1 root systemd-journal  8388608 Oct 23 19:56 user-1014.journal
-rw-r-----+ 1 root systemd-journal  8388608 Oct 24 14:14 user-1015.journal
However, I have some files that are created in /run/systemd/journal

Code: Select all

❯ ls -la /run/systemd/journal/
total 0
drwxr-xr-x  2 root root 120 Feb  1 15:44 .
drwxr-xr-x 19 root root 460 Feb  1 15:43 ..
srw-rw-rw-  1 root root   0 Feb  1 15:44 dev-log
srw-rw-rw-  1 root root   0 Feb  1 15:44 socket
srw-rw-rw-  1 root root   0 Feb  1 15:44 stdout
srw-rw-rw-  1 root root   0 Feb  1 15:43 syslog

Post Reply