[Solved] System stopped booting due to systemd failing to start some services, can't boot into recovery mode either

Off-Topic discussions about science, technology, and non Debian specific topics.
Post Reply
Message
Author
dexcy
Posts: 7
Joined: 2024-11-09 11:47
Has thanked: 1 time

[Solved] System stopped booting due to systemd failing to start some services, can't boot into recovery mode either

#1 Post by dexcy »

My system stopped booting suddenly and I can't figure out the cause (have been using the system for quite some time just fine).Here's what it says during boot

Code: Select all

failed to start postgresql@14-main.service postgresql cluster 14-main
failed to start postgresql@15-main.service postgresql cluster 15-main
failed to start vmware.service lsd: this service starts and stops vmware services
failed to start docker.service - docker application container engine
failed to start snapd. service - snap daemon
Sometimes several of these lines appear (like 4 or 5 for docker), sometimes only postgresql@14 and not both versions (I don't remember why I installed two versions but it didn't interfere with any normal functionality).After these lines nothing happens, and I have nothing to do but to shut my ThinkpadT490 down. Now when I boot again I try to go into recovery mode in GRUB. It then asks me for my encryption password (just like during normal boot), which I successfully enter. Then some "OK" logs and then...

Code: Select all

[FAILED] Failed to start systemd-update-utmp.service - Record System Boot/Shutdown in UTMP
 [DEPEND] Dependency failed for systemd-update-utmp.service - Record System Boot/Shutdown in UTMP
Then some more normal logs, then "Cannot open access to console, the root account is locked, please enter to continue" -> "reloading system manager configuration; starting default.target", some normal logs, and then it's just stuck. No access to console, no choice but to shut down again. What do I do? I get that something seemingly screwed up with systemd but how and why (and why these particular services)? I was using the system normally pretty much this morning, didn't do anything potentially damaging besides some Internet browsing, and in the evening this happens. I don't think I ran any updates during last session as well. Only problematic things with my system I can think of over the last week is several days ago when I ran out of storage space during compiling (I guess swap was used and then I cleaned up a bit of unnecessary files and it seemingly didn't have any immediate effect on the system - I never ran out of space during an ongoing process that takes up gigabytes of space, though, so it was a useful experience in a sense, I guess?). Also haven't run updates in a long time and updated firefox a few days ago, but there weren't much dependencies iirc, certainly nothing related to docker, vmware, utmp and postgres. And again, it didn't affect the boot (why should it, indeed). Oh, and the last shutdown (before all these problems popped up) took longer than usual, a couple of minutes compared to the usual few seconds. Any suggestions and help would be appreciated.
Last edited by dexcy on 2024-11-16 20:49, edited 1 time in total.

Aki
Global Moderator
Global Moderator
Posts: 4093
Joined: 2014-07-20 18:12
Location: Europe
Has thanked: 117 times
Been thanked: 544 times

Re: System stopped booting due to systemd failing to start some services, can't boot into recovery mode either

#2 Post by Aki »

Hello,

What is your installed Debian version ?

Can you record a video of the boot sequence booting into single user mode ?
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄⠀

dexcy
Posts: 7
Joined: 2024-11-09 11:47
Has thanked: 1 time

Re: System stopped booting due to systemd failing to start some services, can't boot into recovery mode either

#3 Post by dexcy »

Aki wrote: 2024-11-09 12:00 Hello,

What is your installed Debian version ?

Can you record a video of the boot sequence booting into single user mode ?
I have a Debian-derivative system, Kali (needed it for coursework studies), though I don't think the problem is anything Kali-specific

From cat /proc/version:

Code: Select all

linux version 6.1.0-kali9-amd64 (devel@kali.org) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian 6.1.27-1kali1 (2023-05-12)


Here's a video of the boot process in recovery/single user mode:

https://drop.infini.fr/r/MN5UYTG1gA#46c ... y1QRkzrYE=

Aki
Global Moderator
Global Moderator
Posts: 4093
Joined: 2014-07-20 18:12
Location: Europe
Has thanked: 117 times
Been thanked: 544 times

Re: System stopped booting due to systemd failing to start some services, can't boot into recovery mode either

#4 Post by Aki »

dexcy wrote: 2024-11-09 12:40
Aki wrote: 2024-11-09 12:00 What is your installed Debian version ?
Can you record a video of the boot sequence booting into single user mode ?
I have a Debian-derivative system, Kali (needed it for coursework studies), though I don't think the problem is anything Kali-specific
From cat /proc/version:

Code: Select all

linux version 6.1.0-kali9-amd64 (devel@kali.org) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian 6.1.27-1kali1 (2023-05-12)

Here's a video of the boot process in recovery/single user mode:
https://drop.infini.fr/r/MN5UYTG1gA#46c ... y1QRkzrYE=
The screen is not understandable in many moments, but as you seem to be using separate partitions for some mount points (e.g. /var), it is possible that one partition (e.g. the /var partition) is full.

You can boot using a live disk to check it out.

Hope this helps.

--
note: moved to "Off-Topic" sub-forum because the post is about a derivative distribution.
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄⠀

dexcy
Posts: 7
Joined: 2024-11-09 11:47
Has thanked: 1 time

Re: System stopped booting due to systemd failing to start some services, can't boot into recovery mode either

#5 Post by dexcy »

Aki wrote: 2024-11-09 13:01
dexcy wrote: 2024-11-09 12:40
Aki wrote: 2024-11-09 12:00 What is your installed Debian version ?
Can you record a video of the boot sequence booting into single user mode ?
I have a Debian-derivative system, Kali (needed it for coursework studies), though I don't think the problem is anything Kali-specific
From cat /proc/version:

Code: Select all

linux version 6.1.0-kali9-amd64 (devel@kali.org) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian 6.1.27-1kali1 (2023-05-12)

Here's a video of the boot process in recovery/single user mode:
https://drop.infini.fr/r/MN5UYTG1gA#46c ... y1QRkzrYE=
The screen contents are understandable in many parts, but because it seems you are using separate partitions for some mount points (i.e. /var) it is possible that some partition is full (i.e. the /var partition).

You can boot using a live disk to check it out.

Hope this helps.

--
note: moved to "Off-Topic" sub-forum because the post is about a derivative distribution.
I recall checking free space recently and /var had around 1 GB though. Even if it was full, is it possible my problems are caused by such a thing?

UPD: I can get shell access if I add rw init=/bin/bash parameter in boot options. Though if I understand correctly it boots without systemd so I can't really use any systemctl commands. And most partitions aren't mounted.
Last edited by dexcy on 2024-11-09 13:19, edited 1 time in total.

Aki
Global Moderator
Global Moderator
Posts: 4093
Joined: 2014-07-20 18:12
Location: Europe
Has thanked: 117 times
Been thanked: 544 times

Re: System stopped booting due to systemd failing to start some services, can't boot into recovery mode either

#6 Post by Aki »

Hello,

You can boot using a Debian Live ISO [1] and check for the status of different partitions and their file systems.

How did you partition your disk ?

How many partition ?

For what mount points ?

Did you create a separate partition for /etc ?

Hope this helps.


[1] https://www.debian.org/CD/live/
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄⠀

dexcy
Posts: 7
Joined: 2024-11-09 11:47
Has thanked: 1 time

Re: System stopped booting due to systemd failing to start some services, can't boot into recovery mode either

#7 Post by dexcy »

Aki wrote: 2024-11-09 16:26 Hello,

You can boot using a Debian Live ISO [1] and check for the status of different partitions and their file systems.

How did you partition your disk ?

How many partition ?

For what mount points ?

Did you create a separate partition for /etc ?

Hope this helps.


[1] https://www.debian.org/CD/live/
Here's partition info (sorry for the photos from rescue iso, visibility is fine though):

https://drop.infini.fr/r/GcUYV23EbR#BSI ... S3ShIV20c=

No separate partition for /etc

Aki
Global Moderator
Global Moderator
Posts: 4093
Joined: 2014-07-20 18:12
Location: Europe
Has thanked: 117 times
Been thanked: 544 times

Re: System stopped booting due to systemd failing to start some services, can't boot into recovery mode either

#8 Post by Aki »

Hello,

Are you using a Debian Live ISO or a Debian Install ISO for the troubleshooting ?

Is it you cross-posting ? Have you cross-posted on any other forums ?

Use the Debian ISO (started in rescue mode) to try to mount the decrypted /dev/mapper/Debian-var and /dev/mapper/Debian-tmp partitions and explore their contents and used/free space; e.g.:

Code: Select all

mkdir /my_var
mkdir /my_tmp
mount /dev/mapper/Debian-var /my_var
mount /dev/mapper/Debian-tmp /my_tmp
df -h
cd /my_var
du -sh
cd /my_tmp
du -sh
Hope this helps.
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄⠀

dexcy
Posts: 7
Joined: 2024-11-09 11:47
Has thanked: 1 time

Re: System stopped booting due to systemd failing to start some services, can't boot into recovery mode either

#9 Post by dexcy »

Aki wrote: 2024-11-10 14:31 Hello,

Are you using a Debian Live ISO or a Debian Install ISO for the troubleshooting ?

Is it you cross-posting ? Have you posted on any other forums ?

Use the Debian ISO (started in rescue mode) to try to mount the decrypted /dev/mapper/Debian-var partition and explore its contents and used/free space; e.g.:

Code: Select all

mkdir /my_var
mkdir /my_tmp
mount /dev/mapper/Debian-var /my_var
mount /dev/mapper/Debian-tmp /my_tmp
df -h
cd /my_var
du -sh
cd /my_tmp
du -sh
Hope this helps.
I used this one:
https://cdimage.debian.org/debian-cd/cu ... etinst.iso

Yes, it's me cross-posting because forums are slow and I needed advice, not relying only on one place. Is it wrong? I appreciate your answers, I should have thought of doing what you describe earlier.

du is not part of the rescue mode utils apparently (I can use awk to check filesize though) but even with just dh I see that /dev/mapper/Debian-var is 100% full (tmp is mostly free, only 88Kb out of 1.8 Gb used, compared to 9.1Gb our of 9.1 Gb for var). That's not good. Do you think it can be the primary reason (and why, if so?).

P.S. I'm doing all that without using a root file system. Is it ok or should I better choose a device to use as root filesystem (like /dev/Debian/var, /dev/Debian/root and so on?)
Last edited by dexcy on 2024-11-10 16:21, edited 2 times in total.

Aki
Global Moderator
Global Moderator
Posts: 4093
Joined: 2014-07-20 18:12
Location: Europe
Has thanked: 117 times
Been thanked: 544 times

Re: System stopped booting due to systemd failing to start some services, can't boot into recovery mode either

#10 Post by Aki »

Hello,
dexcy wrote: 2024-11-10 15:51 Yes, it's me cross-posting because forums are slow and I needed advice, not relying only on one place. Is it wrong?
Cross-posting is discouraged, as it can result in wasting the limited time resources of several uncoordinated volunteers on a topic that each single volunteer is unaware of. Please avoid this in the future.
dexcy wrote: 2024-11-10 15:51 [..] I see that /dev/mapper/Debian-var is 100% full [..] Do you think it can be the primary reason (and why, if so?).
Yes, this may be the cause. The operating system hosts many files in the /var tree that are required for proper operation.
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄⠀

Aki
Global Moderator
Global Moderator
Posts: 4093
Joined: 2014-07-20 18:12
Location: Europe
Has thanked: 117 times
Been thanked: 544 times

Re: System stopped booting due to systemd failing to start some services, can't boot into recovery mode either

#11 Post by Aki »

Hello,
dexcy wrote: 2024-11-10 15:51 P.S. I'm doing all that without using a root file system. Is it ok or should I better choose a device to use as root filesystem (like /dev/Debian/var, /dev/Debian/root and so on?)
The Debian installer in rescue mode has its own (limited) root file system. You can use it to list the contents of /dev/mapper/Debian-var, identify some unnecessary files and delete them to free up some space to boot the operating system.

If you are booting from a Debian Live-ISO, you can run a full read-only Debian distribution for maintenance purposes to achieve the same result, but with a full desktop environment at your disposal.

Hope this helps. Please let me know.
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄⠀

dexcy
Posts: 7
Joined: 2024-11-09 11:47
Has thanked: 1 time

Re: System stopped booting due to systemd failing to start some services, can't boot into recovery mode either

#12 Post by dexcy »

Aki wrote: 2024-11-10 18:11 Hello,
dexcy wrote: 2024-11-10 15:51 P.S. I'm doing all that without using a root file system. Is it ok or should I better choose a device to use as root filesystem (like /dev/Debian/var, /dev/Debian/root and so on?)
The Debian installer in rescue mode has its own (limited) root file system. You can use it to list the contents of /dev/mapper/Debian-var, identify some unnecessary files and delete them to free up some space to boot the operating system.

If you are booting from a Debian Live-ISO, you can run a full read-only Debian distribution for maintenance purposes to achieve the same result, but with a full desktop environment at your disposal.

Hope this helps. Please let me know.
Well, you were right. I found a pretty big .partial snap package in /var/lib/snapd directory dated the same day I started having problems...I guess snap decided to autoupdate it and filled /var to the limit, at least that's my working theory. After cleaning up a bit I booted successfully, and no messages about failing services greeted me. Sometimes solutions can be simple...thank you :)

Aki
Global Moderator
Global Moderator
Posts: 4093
Joined: 2014-07-20 18:12
Location: Europe
Has thanked: 117 times
Been thanked: 544 times

Re: System stopped booting due to systemd failing to start some services, can't boot into recovery mode either

#13 Post by Aki »

Hello,
dexcy wrote: 2024-11-10 20:33 Well, you were right. I found a pretty big .partial snap package in /var/lib/snapd directory dated the same day I started having problems...I guess snap decided to autoupdate it and filled /var to the limit, at least that's my working theory. After cleaning up a bit I booted successfully, and no messages about failing services greeted me. Sometimes solutions can be simple...thank you :)
I'm glad you sorted it out. :)

Please, mark the discussion as "solved" manually adding the text tag "[Solved]" at the beginning of the subject of the first message.
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄⠀

Post Reply