[SOLVED]Help troubleshooting when suspending to S3 with Self Encrypting SSD (sedutil)

[asana]

System: Debian Stable 12 x86_64
Old notebook Intel Core™ i7-2720QM and Nvidia GeForce® GTX 560M, and I'm using a self encrypting SSD configured with sedutil.
I boot with the parameter libata.allow_tpm=1 and I have a systemd service to store the password in the kernel allowing it to decrypt the disk on waking up.

Problem: since linux-image-6.1.0-14-amd64 (through 15, 16, 17 and 18) the system is unable to fully wake up from suspend. Most of the time it wakes up to a black screen and CAPS LOCK led doesn't change when pressing the CAPS LOCK button. Sometimes the monitor turns on and I can login in a tty but no command ever works. Not even `reboot` `shutdown` etc. Regardless if the monitor turns on, I can shutdown with Alt + SysRq + B.

(bug reports: 1058028, 1058052, 1058890)

What I've tested:

1) Adding: nomodeset, init_on_alloc=0, intel_iommu=off to boot the kernel. No change at all.

2) I tested suspend by writing to /sys/power/pm_test the following successively "freezer", "devices", "platform", "processors", "core". Every single test was successful, ie, the system wakes up on its own after 5 seconds.

3) `echo mem > /sys/power/state"` instead of `systemctl suspend`. No change.

4) Changing /etc/systemd/sleep.conf to have "SuspendState=standby" or "SuspendState=freeze" seems to make things worse because the notebook seems unable to enter sleep mode, while "SuspendState=mem" causes the same behavior as reported here.

5) I recompiled sedutil after booting linux-image-6.1.0-18-amd64 (git clone --branch s3-sleep-support https://github.com/badicsalex/sedutil.git). It made no difference.

EDIT: it is a Linux Kernel bug (Regression): 218538

[dashek]

@asana,
have you read last messages of system journal after having this problem? (You can see it with "journalctl -b -1", meaning "select boot -1 from current".) I mean messages that have been logged after waking up from suspend. Are these present in journal at all? (You should be able to figure it out from messages' timestamps.)
Also, does "dmesg" work when you manage to log into tty after such event?

AFAIK, kernel produces at least couple of journal entries after every wake up. If these get saved to your disk before rebooting, there is no problem with disk encryption.

[Aki]

Hello @asana,

[..]
I'm using a self encrypting SSD configured with sedutil
[..]
Problem: since linux-image-6.1.0-14-amd64 (through 15, 16, 17 and 18) the system is unable to fully wake up from suspend.
[..]

It seems to me that the sedutil programm is not in the Debian repositories.

It claims to be a copyrighted program distributed under the GPL licence, but there is no LICENSE file in its git repository.

I've never used it, and I'm not currently motivated to delve into its inner workings. However, if you know about it, please share your knowledge in the forum.

The "README" file [1] in the git repository you quoted in the previous message says:

***** sleep (S3) is not supported.

Therefore, it seems the S3 in not supported by design.

Sorry I cannot be more helpful.

Please, mark the discussion as "solved" manually adding the text tag "[Solved]" at the beginning of the subject of the first message (after other tags, if any); for example:

[Solved] Debian 12: cannot suspend to S3 booting with sedutil

—
[1] https://github.com/badicsalex/sedutil/b ... /README.md

[asana]

@Aki I'm not sure if it is a problem with the disk self encryption. Even though "official" sedutil doesn't support suspend, the fork I'm using does support it. I'm planning on trying another disk to figure it out and then I post here. (Also Linux kernel has a module to make it work since 4.13 if I remember it right)

[asana]

@dashek I can't find in journal any cue any worth mentioning error. Is there any word I should pay attention to? (https://bugs.debian.org/cgi-bin/bugrepo ... 1058052#22) I will try to check if dmesg works, thank you so much!

[asana]

I didn't manage to get to a tty again, but this is the dmesg before going to sleep: app.php/pastebin/?mode=view&s=91 and this is the journalctl -b -1 app.php/pastebin/?mode=view&s=90

[asana]

I tried to install in another disk, but the disk is faulty. I will try in another disk asap.

[asana]

I'm still not completely convinced the self encrypting drive is the culprit. But I still don't have another disk drive to test.
Meanwhile I managed to get into a terminal after partially getting back from suspend. dmesg doesn't work as almost every command I test doesn't. But /opt/sedutil-cli does seems to work, I tried to unlock the device and it prints a message saying it is unlocked.
If anyone has any idea on troubleshooting, I would really appreciate it.
I can keep using an older kernel, that works flawless, but I want to understand what the problem is.

[Aki]

[..]
If anyone has any idea on troubleshooting, I would really appreciate it.
I can keep using an older kernel, that works flawless, but I want to understand what the problem is.

You can boot replacing the "quiet" kernel boot parameter with the "debug" kernel boot parameter: it will make your kernel more verbose. See:

• Configuring GRUB v2

Then, you can follow directions from kernel documentation about:

• Debugging hibernation and suspend

Then, you can try suspending after the OS is booted into the multi-user.target (instead of graphic.target); see:

• https://forums.debian.net/viewtopic.php?p=787976#p787976

You can check if the system responds to the REISUB sequence:

• Magic Sys Req

All the above should allow you to collect more clues in log files or screen output.

EDIT: I would suggest to include in the subject of the first post that you are using a self encrypting SSD configured with sedutil; for example:

Help troubleshooting when suspending to S3 with self encrypting SSD (sedutil)

--
note: moved from "System and Network configuration" to "General Question" sub-forum.

[asana]

Then, you can follow directions from kernel documentation about:
Debugging hibernation and suspend

Every test goes fine. The system suspend and wakes up on its own after 5 seconds just fine.

ou can try suspending after the OS is booted into the multi-user.target

It doesn't change anything.

Magic Sys Req

The only Sys Req that seems to work is Alt + SysRq + B

The Linux kernel is reported to work well with sedutil at least up to 6.6.7-arch1-1 (Arch Linux) by the autor of the fork with S3 suspend support.
Either the problem is not related to SED, or it is a specific Debian patch.

[Aki]

Hello,

[..]

Magic Sys Req

The only Sys Req that seems to work is Alt + SysRq + B
[..]

It means that the kernel is not completely stuck, since it can be rebooted with the previous sequence.

Is there something more in system logs shutting down with the REISUB sequence ?

Did you configure the kernel to boot with the debug kernel parameter ?

If the computer appears to be frozen, have you tried logging in using an ssh client (you need to install the ssh server first, of course)?

[lindi]

Does suspend with with debian 12 live usb? This should be a way to rule out problems caused by storage devices.

[dashek]

Some thoughts:
1)

I didn't manage to get to a tty again, but this is the dmesg before going to sleep: app.php/pastebin/?mode=view&s=91 and this is the journalctl -b -1 app.php/pastebin/?mode=view&s=90

After reading your logs, I'm almost sure that the problem is with disk access. This would explain why there is no trace of messages generated after wake-up*. Tha fact that only some programs work may be explained with them being already present in the memory, so they can be run with no need to fetch them from disk.

* it can also be result of Alt+SysRq+b. Wikipedia describes it as:

Immediately reboot the system, without unmounting or syncing filesystems

so journal entries may be lost before actual writing to the disk when you press this key combination. The filesystem is 'up to date' only in memory, and is 'synced' with hardware only when kernel decides to do so.

2) The last commit in branch you're using is about 2 years old. The Drive-Trust-Alliance/sedutil:master also hasn't received any important updates for years. Maybe sedutil does not follow changes included in linux 6.1.0-14. It can mean that kernel update broke some kind of backward compatibility, what can be considered a bug. But sometimes, certain kernel features are announced to be changed or removed. I such changes are announced long before applying them.

[asana]

@dashek

After reading your logs, I'm almost sure that the problem is with disk access. This would explain why there is no trace of messages generated after wake-up*. Tha fact that only some programs work may be explained with them being already present in the memory, so they can be run with no need to fetch them from disk.

I think you are right on that.

sometimes, certain kernel features are announced to be changed or removed. I such changes are announced long before applying them.

But everything is supposed to work fine up to at least Linux 6.6.7-arch1-1 (Arch Linux), reported by the author of the fork of the sedutil I use.

I wonder if it may be a bug in one of the patches applied by the Linux Kernel team of Debian.

I will try to install Debian in an unencrypted disk asap to test.

[asana]

@Aki I changed just the disk of the notebook and installed Debian 12 on it, without any encryption and suspend works fine on kernel 6.1.0-18. Which makes me conclude that something is wrong with some patch of that kernel, not allowing the disk to be decrypted with sedutil after waking up.

I booted with `debug`, and I couldn't see any different message from my normal boot (which isn't quiet, it is just normal text mode). The journalctl is 93 .

[Aki]

@Aki I changed just the disk of the notebook and installed Debian 12 on it, without any encryption and suspend works fine on kernel 6.1.0-18. Which makes me conclude that something is wrong with some patch of that kernel [..]

You conclusion cannot be confirmed unless you verify it with a "vanilla"/upstream kernel (same version 6.1.76) compiled with the same configuration of the Debian kernel (without Debian patches).

[asana]

I compiled the kernel 6.1.76 from Debian sources without applying any patch and it can't wake up from suspend I don't know where to go from here.

[Aki]

I compiled the kernel 6.1.76 from Debian sources without applying any patch and it can't wake up from suspend :-( I don't know where to go from here.

Since the REISUB sequence can still reboot the machine after resume from suspension, this means that the kernel is running after resume. The OS is not completely stuck.

It is possible that some hardware / software components are having trouble restarting after resume: you need to identify them (if any).

Are wired network adapter leds blinking after resume ? If yes, you could try connecting using a ssh client to access system logs remotely.

You can also boot Debian in multi-user.target or rescue.target (instead of default.target) and try suspending from here (to exclude loading some operative systems components before suspending) using the command:

Code: Select all

systemctl suspend

As an example, see:

• https://manpages.debian.org/bookworm/systemd/systemd.target.5.en.html

Hope this helps.

[asana]

Reported the bug against Linux kernel at bugzilla 218538.

[asana]

Turns out it is an upstream Linux Kernel Regression, I did a bisection and found the bad commit.