[Software] Kernel lockup

[segviolate]

I realize this is a bit general, but have there been any reports of kernel lockups with Debian 11? I have a system that runs fine on Debian 10, but I'm experiencing random lockups running it on Debian 11. There isn't a kernel panic displayed, everything becomes unresponsive and I have to do a hard reboot of the machine. Examining the logs post reboot shows no indication of an issue. The system is using kernel 5.10.0-21-amd64, on an Intel i5-8500 and Q370 chipset.

Barring any reports of this issue, I'm looking for ideas as to how I can wedge myself into the system when it's in this state, so that I can get some useful information.

[Aki]

Hello,
You can try the REISUB sequence [1] to shutdown the operative system in a controlled way. This could allow you to save data in the system log before the shutdown.

[1] https://en.wikipedia.org/wiki/Magic_SysRq_key

[None1975]

Hello,
You can try the REISUB sequence

Magic keys tend to be disabled in Debian these days, so you can't just hard-reboot your machine or kill all your X processes by pressing a few keys accidentally.

[Aki]

Hello,

Hello,
You can try the REISUB sequence

Magic keys tend to be disabled in Debian these days, so you can't just hard-reboot your machine or kill all your X processes by pressing a few keys accidentally.

Thank you for your remarks.

According to [1]:

Since the Etch release, the Magic SysRq key feature is enabled in the Linux kernel to allow console users certain privileges. You can confirm this by checking if the /proc/sys/kernel/sysrq exists and reviewing its value:

Code: Select all

$ cat /proc/sys/kernel/sysrq 
438

For example, on my Debian Bullseye (stable, 11.6) (I'm currently running 5.10.0-16-amd64, I know, it's not the latest version for current stable):

Code: Select all

$ cat /proc/sys/kernel/sysrq
438

According to [2] the bit mask in /proc/sys/kernel/sysrq is:

Code: Select all

  2 =   0x2 - enable control of console logging level
  4 =   0x4 - enable control of keyboard (SAK, unraw)
  8 =   0x8 - enable debugging dumps of processes etc.
 16 =  0x10 - enable sync command
 32 =  0x20 - enable remount read-only
 64 =  0x40 - enable signalling of processes (term, kill, oom-kill)
128 =  0x80 - allow reboot/poweroff
256 = 0x100 - allow nicing of all RT tasks

Decimal 438 equals binary 110110110 , therefore are active by default in Debian the following SYSRQ features:

Code: Select all

  2 =   0x2 - enable control of console logging level
  4 =   0x4 - enable control of keyboard (SAK, unraw)
 16 =  0x10 - enable sync command
 32 =  0x20 - enable remount read-only
128 =  0x80 - allow reboot/poweroff
256 = 0x100 - allow nicing of all RT tasks

These should be enough to "gently" (as far as possible) shutdown the system (if the kernel responds to the SYSRQ sequence).

If you want to enable on the fly the dumps of processes:

Code: Select all

  8 =   0x8 - enable debugging dumps of processes etc.

it is possible to issue the command:

Code: Select all

su -l -c "echo 446 > /proc/sys/kernel/sysrq"

or modify /etc/sysctl.conf and update the initramfs.

---
[1] https://www.debian.org/doc/manuals/secu ... rq.it.html
[2] https://www.kernel.org/doc/html/latest/ ... sysrq.html

[Aki]

@segviolate: besides the REISUB sequence, you can check if the terminal console is active and, if yes, you can try to enter there and issue some commands to shutdown the system and/or read system logs to understand what's happening. You can activate the terminal console pressing simultaneously the keys CTRL+ALT+F2 , (e.g. terminal console number 2).

[segviolate]

I've tried the alternate terminal via alt - F[0-9] but I'm not able to get another terminal session.

[Aki]

Hello,

I've tried the alternate terminal via alt - F[0-9] but I'm not able to get another terminal session.

If the graphical environment is running, the key sequence to activate the console terminal is CTRL-ALT-F[0-9].

Have you tried the REISUB sequence ?

[segviolate]

If the graphical environment is running, the key sequence to activate the console terminal is CTRL-ALT-F[0-9].

Have you tried the REISUB sequence ?

I mis-typed and forgot the ctrl. I haven't had a lockup since I learned of the Magic SysRq key sequence. It will be the first thing I try if it happens again.

[CynicalDebian]

Usually lockups or freezes are because of running out of memory, it might be good to leave 'top' running in the background to see if you can spot a memory hog culprit. If you aren't experienced with top, 'htop' is a prettier version that I find easier to understand that is available in the repos.

[Aki]

Usually lockups or freezes are because of running out of memory [..]

In this case, you should see the disk continuously spinning if ram swap is configured.

[sunrat]

Examining the logs post reboot shows no indication of an issue.

Which logs were you looking at?

Code: Select all

journalctl -b -1 -p 3

should show errors for the previous boot. Use -p 4 to show warnings as well, or omit -p for the full journal log.

Hello,

I've tried the alternate terminal via alt - F[0-9] but I'm not able to get another terminal session.

If the graphical environment is running, the key sequence to activate the console terminal is CTRL-ALT-F[0-9].

Have you tried the REISUB sequence ?

If you want to switch to a text terminal, use F2-F6. F7-F12 are Xserver instances, typically default is F7. A good reference is at - https://calcoen.web.cern.ch/Linux_keys.htm

Magic SysReq keys (eg. REISUB) can be reconfigured as well but 438 should be enough. The value of 438 is obtained from the sum of 2 + 4 + 16 + 32 + 128 + 256, so all the corresponding functions are enabled.

[Aki]

@segviolate: have you any update on the topic ?