[Solved] Systemd hanging on last part of shutdown requiring power button press.

[FreeBird]

Fresh install of Debian 12 on Lenovo W541
system specs:
UEFI BIOS Version: GNET($WW (2.42)
Embedded Controller Version: GMHT25WW (1.10)
ME Firmware Version 9.1.20.1035
Machine Type Model 20EFCT01WW
CPU Type: Intel Core i7-4810MQ @2.8GHZ

when initiating a system shutdown, systemd stops/closes all the processes, blanks the screen, displays the following:

Broadcast message from user@W541-debian (date time):
The system will power off now!

Code: Select all

[  48.134884] systemd-shutdown[1]: Waiting for process: 387 ((udev-worker)), 356 ((udev-worker)), 350(systemd-udevd), 585
[128.131373] systemd-shutdown[1]: sending SIGKILL to remaining processes...
[128.138277]  systemd-shutdown[1]: sending SIGKILL to PID 350 (systemd-udevd).
[128.138591] systemd-shutdown[1]: sending SIGKILL to PID 356 ((udev-worker)).
[128.138817] systemd-shutdown[1]: sending SIGKILL to PID 379 ((udev-worker)).
[128.138977] systemd-shutdown[1]: sending SIGKILL to PID 387 ((udev-worker)).
[128.139447] systemd-shutdown[1]: sending SIGKILL to PID 585 ((udev-worker)).
[138.140382] systemd-shutdown[1]: waiting for process: 585 (modprobe).
[218.141675] systemd-shutdown[1]: Unmounting file systems.
[218.143578] (sd-remount) [4334]: remounting '/' read-only with options 'errors=remount-ro'.
[218.156368] EXT4-fs (dm-0): re-mounted. Quota Mode: none.
[218.163836]systemd-shutdown[1]: All filesystems unmounted.
[218.163853] systemd-shutdown[1]:Deactivating swaps.
[218.163903] systemd-shutdown[1]:All swaps deactivated.
[218.163912] systemd-shutdown[1]:Detaching loop devices.
[218.164390] systemd-shutdown[1]:All loop devices detached.
[218.164404] systemd-shutdown[1]:Stopping MD devices.
[218.164465] systemd-shutdown[1]:All MD devices stopped.
[218.164476] systemd-shutdown[1]:Detaching DM devices.
[218.164773] systemd-shutdown[1]:Detaching DM /dev/dm-1 (254:1).
[218.220390] systemd-shutdown[1]:Not all DM devices detached, 1 left.
[218.220490] systemd-shutdown[1]:Detaching DM devices.
[218.220755] systemd-shutdown[1]:Not all DM devices detached, 1 left.
[218.220800] systemd-shutdown[1]::Detaching DM devices.
[218.220974] systemd-shutdown[1]:Not all DM devices detached, 1 left.
[218.220989] systemd-shutdown[1]:Cannot finalize remaining DM devices, continuing.
[218.283318] systemd-shutdown[1]:Failed to finalize DM devices, ignoring.
[218.283517] systemd-shutdown[1]:Syncing filesystems and block devices.
[218.283795] systemd-shutdown[1]:Powering off.

At his point the system remains powered on until a long press on the power button.

upon rebooting and checking the systemd journal with journalctl -r the following entries are listed at the point of shut down and reboot:
..
..
app.slice: consumed 3min 2.254s CPU time.
reached target shutdown.target -shutdown.
Finished systemd-exit.service - Exit the Session.
Reached target exit.target - Exit the Session.
--- Boot 21d78##########...##### ---
Queued start job for default target default.target.
..
..
Any ideas will be greatly appreciated.

[CwF]

Please put large code blocks in code tags as I did for one of the sections. Re-open the message with 'edit' to see the example.

[FreeBird]

Got it

Code: Select all

.........

thanks for the tip.

[bin]

Please provide the output from

Code: Select all

 inxi -SMCGx

and also from

Code: Select all

lsblk

Make sure you use the code tags.
Once we know what dm-1 is then it should be possible to work put what's going on. Do you keep an external devices connected all the time?

[FreeBird]

bin,
hope this helps. Thank you.

Code: Select all

jon@W541-debian:~$ inxi -SMCGx
System:
  Host: W541-debian Kernel: 6.1.0-18-amd64 arch: x86_64 bits: 64 compiler: gcc
    v: 12.2.0 Desktop: KDE Plasma v: 5.27.5 Distro: Debian GNU/Linux 12
    (bookworm)
Machine:
  Type: Laptop System: LENOVO product: 20EFCTO1WW v: ThinkPad W541
    serial: <superuser required>
  Mobo: LENOVO model: 20EFCTO1WW serial: <superuser required> UEFI: LENOVO
    v: GNET94WW (2.42 ) date: 06/02/2021
CPU:
  Info: quad core model: Intel Core i7-4810MQ bits: 64 type: MT MCP
    arch: Haswell rev: 3 cache: L1: 256 KiB L2: 1024 KiB L3: 6 MiB
  Speed (MHz): avg: 800 min/max: 800/3800 cores: 1: 800 2: 800 3: 800 4: 800
    5: 800 6: 800 7: 800 8: 800 bogomips: 44698
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3
Graphics:
  Device-1: Intel 4th Gen Core Processor Integrated Graphics vendor: Lenovo
    driver: i915 v: kernel arch: Gen-7.5 bus-ID: 00:02.0
  Device-2: NVIDIA GK107GLM [Quadro K1100M] vendor: Lenovo driver: nouveau
    v: kernel arch: Kepler bus-ID: 01:00.0 temp: 47.0 C
  Device-3: Lite-On Integrated Camera type: USB driver: uvcvideo
    bus-ID: 2-12:5
  Display: wayland server: X.Org v: 1.22.1.9 with: Xwayland v: 22.1.9
    compositor: kwin_wayland driver: dri: crocus gpu: i915
    resolution: 1920x1080~60Hz
  API: OpenGL v: 4.6 Mesa 22.3.6 renderer: Mesa Intel HD Graphics 4600 (HSW
    GT2) direct-render: Yes
    

Code: Select all

jon@W541-debian:~$ lsblk
NAME                        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda                           8:0    0 238.5G  0 disk 
├─sda1                        8:1    0   512M  0 part /boot/efi
├─sda2                        8:2    0   488M  0 part /boot
└─sda3                        8:3    0 237.5G  0 part 
  ├─W541--debian--vg-root   254:0    0  23.3G  0 lvm  /
  ├─W541--debian--vg-var    254:1    0   9.3G  0 lvm  /var
  ├─W541--debian--vg-swap_1 254:2    0   976M  0 lvm  [SWAP]
  ├─W541--debian--vg-tmp    254:3    0   1.9G  0 lvm  /tmp
  └─W541--debian--vg-home   254:4    0 202.1G  0 lvm  /home
sr0                          11:0    1  1024M  0 rom  

[Aki]

Hello,

Could you boot Debian into multi-user.target or rescue.target (instead of default.target) and try to shutdown from there (to avoid loading some operating system components before shutdown) ? You can do it adding systemd.unit=rescue.target or just 1 (the number one) to the kernel command line. [1]

After booting as described above, you can shutdown from the command line using the command:

Code: Select all

shutdown -h now

It would be interesting to know the result of this test.

--
[1] Debian Wiki - Systemd hangs on startup or shutdown

[FreeBird]

Aki,
Thank you for responding to my problem with systemd during shutdown. Please bear with me as I try and get up to speed on systemd. I have spent the last 4 hours trying to learn about the boot process BIOS Vs UEFI/ GRUB2 / Kernel loading/ and systemd initialization. I read the following: https://wiki.debian.org/systemd#systemd ... r_shutdown and https://wiki.ubuntu.com/Kernel/KernelBootParameters and https://ubuntu.com/core/docs/modify-kernel-options and https://freedesktop.org/wiki/Software/s ... Debugging/

I am able to enter the GRUB command line by pressing "e" during GRUB splash screen. I am not sure exactly what "adding systemd.unit=rescue.target or just 1 (the number one) to the kernel command line." Means. I tried typing it into the "grub>" command line and nothing. so obviously I am missing something. Is there some other command line during boot up? Is there a "script" file I need to edit? if so, where is the location of this file? It seems there is some critical background information I do not have and am eager to learn about.
If you could so graciously point me to another article describing what system function/ file i need to access, learn about to I can perform the function you think will help resolve this issue.
again thank you for bearing with me.

[Aki]

Hello,

If you could so graciously point me to another article describing what system function/ file i need to access, learn about to I can perform the function you think will help resolve this issue.

It will not solve your issue, it's just a test to acquire more information.

There are the steps:
1) on the grub screen (before it boots), press the "e" key: grub will enter "edit" mode (it will allow you to temporarily modify its configuration).
2) Move the cursor with the arrow keys, then move to the end of the line that ends with "quiet" or "ro"
3) add "1" or "2" or "3" (only one of the three without quotes): its meaning is shown in the table below
4) press the F10 key: grub will start booting with the modified configuration
5) at the shell login, type root password (if booted with 1) or root username and password otherwise
6) at the shell prompt as root, enter the command:

Code: Select all

shutdown -h now

7) Check if the system shuts down completely or gets stuck.

A video in webm format (compresses as zip file) is attached for your convenience.

grub2_start_with_runlevel.webm.zip

(277.86 KiB) Downloaded 8 times

Hope this helps.

--
note: these are the meanings of "run levels" for this test:

System V runlevel	systemd target	Purpose
1, S runlevel1.target	rescue.target	Single-user mode
2 runlevel2.target	multi-user.target	Local multiuser without remote network
3 runlevel3.target	multi-user.target	Full multiuser with network

see: Debian Wiki - Systemd Cheat Sheet

[bin]

As near as I can tell from this refrence, https://github.com/systemd/systemd/issues/15004 this is a result of the lvm layout you have. It seems that root / cannot be detached as it is set at the moment.

Just puzzled why you have boot and boot-efi?

TBH I would just bin the lot and start again with a simple:-
sda1 /boot/efi 300mb
sda2 /root 25gb - adjust to your taste
sda3 /home -- the rest

[FreeBird]

Thanks everyone for their help on this issue.
Just a recap. This issue is occurring on two different Thinkpad W541 i just acquired, so fresh installs. I have used the graphical install (Debian12.5.0), selected "Guided- use entire disk and set up LVM", I have tried both " All files in one partition" and " Separate /home, /var, /tmp partitions", I have used the installer on 3 different ssd's on both machines and get the same issue with shutdown hanging at the end. restart, hibernation and sleep do not work either. I did as Aki suggested (thank you for the excellent description /tutorial) and booted systemd under each of three different runlevels, 1, 2, 3. each time I logged in as root and then shutdown. each time the shutdown went great until the last step as described above, doing the exact same waiting for processes udev-worker, modprob, systemd-udevd, sending SIGKILL to the process, and then detaching DM devices. there are slight variations b/n each shutdown. After several minutes of the waiting, killing off process and detaching DM's, It reports "power off", but does not actually perform it and a long press on the power button is required.

since these are fresh installs there is no problem trying installs with different partition configurations. I am not sure how i would go about doing what bin suggested above :
sda1 /boot/efi 300mb
sda2 /root 25gb - adjust to your taste
sda3 /home -- the rest
if someone could describe in a little more detail. I assume I use "Manual" disk configuration.

I appreciate your gracious assistance and suggestions for any other troubleshooting attempts to gain more information on this issue.

[Aki]

Hello,

You can try installing a newer kernel, for example version 6.5 from Debian Backports.

Then, you can try:

• disabling the nvidia graphics card from BIOS (if it is possible) to verify if the computer shuts down

• installing the nvidia proprietary kernel (instead of nouveau you are currently running) to verify if the computer shuts down

Aki.

[FreeBird]

Thank you everyone for helping with this issue.
Aki, thank you for your suggestions.
I installed the 6.5 Kernel headers and 6.5 kernel image from Debian Bookworm Backports using these instructions as well as from synaptic using these instructions .

in both cases the system hung on reboot. in the first case, using apt, it did produce the command prompt. After logging in I was able to verify kernel 6.5 was installed. On shut down it still hung up as described above. Using synaptic to install the 6.5 kernel, the system completely hung up at the end of systemd boot just as it was to launch the display manager login screen. leaving just an unresponsive system with a blank screen.

Looking at BIOS, it does not look possible to disable the nvidia graphics card.

I am not sure where to find the nvidia proprietary kernel. i did not see any here

I installed Debian 11.5.0 on this W541 and Shutdown works great, as it should.

I appreciate any other suggestions for trying other things to troubleshoot this issue with 12.0 Bookworm.

[FreeBird]

SOLVED


Thank You everyone for helping with this issue.
Aki and bin for offering their suggestions, which led me to this page on identifying and installing non-free nvidia drivers from the debian repositories. of course I modified "buster" in the example to "bookworm". After making the entries to the sources.list installing and running nvidia-detect, nvidia-tesla-470-driver was recommended. After installing this and rebooting the system now shutsdown, restarts, hibernates and sleeps perfectly as it should.

SOLVED

[Aki]

Hello,

After making the entries to the sources.list installing and running nvidia-detect, nvidia-tesla-470-driver was recommended. After installing this and rebooting the system now shutsdown, restarts, hibernates and sleeps perfectly as it should.
SOLVED

I'm glad you sorted it out. :)

Happy Debian !