[SOLVED] black screen of death

[Head_on_a_Stick]

But why would gpgsm cause systemd to crash?
.

So you are now running with systemd as PID1?

Do you still have any packages on the box that were build with the no-systemd sources?

Please post the output of

Code: Select all

loginctl session-status --no-p

[Soul Singin']

So you are now running with systemd as PID1?

Yes.

Do you still have any packages on the box that were build with the no-systemd sources?

None.

Please post the output of

Code: Select all

loginctl session-status --no-p

Thank you for taking a look. Here it is:

Code: Select all

2 - eryk (1000)
           Since: Tue 2019-05-14 18:23:49 EDT; 19h ago
          Leader: 1349 (lightdm)
            Seat: seat0; vc7
         Display: :0
         Service: lightdm; type x11; class user
         Desktop: lightdm-xsession
           State: active
            Unit: session-2.scope
                  ├─ 1349 lightdm --session-child 14 23
                  ├─ 1358 x-session-manager
                  ├─ 1414 /usr/bin/dbus-launch --exit-with-session --sh-syntax
                  ├─ 1415 /usr/bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
                  ├─ 1433 /usr/bin/ssh-agent x-session-manager
                  ├─ 1434 icewmbg
                  ├─ 1435 icewm --notify
                  ├─ 1436 icewmtray --notify
                  ├─ 1443 xeyes
                  ├─ 1445 nm-applet
                  ├─ 1448 /usr/lib/at-spi2-core/at-spi-bus-launcher
                  ├─ 1453 /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3
                  ├─ 1455 /usr/lib/at-spi2-core/at-spi2-registryd --use-gnome-session
                  ├─ 1461 clipit
                  ├─ 1467 pnmixer
                  ├─ 1472 /usr/bin/pulseaudio --start --log-target=syslog
                  ├─ 1477 cbatticon
                  ├─ 1515 kdeinit4: kdeinit4 Running...
                  ├─ 1518 kdeinit4: klauncher [kdeinit] --fd=
                  ├─ 1520 kdeinit4: kded4 [kdeinit]
                  ├─ 2862 /usr/bin/kuiserver
                  ├─ 6817 /usr/bin/knotify4
                  ├─17161 /usr/lib/dconf/dconf-service
                  ├─24090 /usr/lib/x86_64-linux-gnu/gconf/gconfd-2
                  ├─28260 dolphin4
                  ├─28280 /usr/bin/firefox-esr
                  ├─28331 /usr/lib/firefox-esr/firefox-esr -contentproc -childID 1 -isForBrowser -intPrefs 235:3| -boolPrefs 36:1|1…ab
                  ├─28396 /usr/bin/expressvpn-browser-helper /home/eryk/.mozilla/native-messaging-hosts/com.expressvpn.helper.json …om
                  ├─28423 /usr/lib/firefox-esr/firefox-esr -contentproc -childID 2 -isForBrowser -intPrefs 235:3| -boolPrefs 36:1|1…ab
                  ├─28624 konsole
                  ├─28627 /bin/bash
                  └─28654 loginctl session-status --no-p

Edit: . After posting, I went back and experimented by installing gpgsm, sending an email with mutt (to trigger usage of gpgsm) and then compared the output of the loginctl command.

The only difference was the PID of the loginctl command.
.

[Soul Singin']

summary:

• The machine is an old Dell Inspiron with an Intel Quad Core i3 CPU and 8 GB of RAM, running Debian Stretch.
• When systemd-sysv is NOT installed, the machine experiences random crashes.
• When systemd-sysv and gpgsm are both installed, the machine also experiences random crashes.
• When gpgsm is NOT installed (and systemd-sysv is installed), the machine runs perfectly.

observations:

• Since the crashes occur when systemd-sysv is not installed, it cannot be systemd-sysv that is causing the crashes. Something else must be creating trouble for the init system.
• gpgsm creates that trouble for the init system even though systemd-sysv is installed.

What is causing those crashes?
.

[sickpig]

can u get gpgsm to do a delayed start?
it will help in identifying if the issue is with init or with the startup of gpgsm
let me know if this approach seems right?

[Soul Singin']

can u get gpgsm to do a delayed start?
it will help in identifying if the issue is with init or with the startup of gpgsm
let me know if this approach seems right?

The approach is correct. I just cannot find a way to implement it.

For example, when I ran the loginctl command, first I ran it on the machine without gpgsm installed and stored the output. Then, I installed the gpgsm package and sent an email with mutt to trigger the gpgsm binary. Finally, I ran the loginctl command again and stored the output.

When I compared the output, the only difference was the PID of the loginctl command -- as you would expect. Or at least, that's what you would expect if no other processes had been started or stopped. . mutt sent the email and so gpgsm had finished too, so those processes had stopped and nothing else was started.

So it's like "nothing" is running, but that "nothing" crashed my machine a short time later.
.

[Head_on_a_Stick]

gpgsm creates that trouble for the init system

What makes you think that PID1 is involved in this problem?

You seem to have a correct login session so my theory was wrong.

Try the gpgsm version from stretch-backports.

Or perhaps install and enable the haveged package to help generate some (fake) entropy.

The systemd-coredump package will store any coredumps for future analysis, see coredumpctl(1).

[Soul Singin']

What makes you think that PID1 is involved in this problem?

The fact that the crash is so catastrophic that it disables everything and does not leave a trace. Not even a log file. I could be mistaken, but it seems to me that the problem is affecting the system at a very deep level.

This morning, I reinstalled gpgsm and sent a bunch of emails with mutt, so I'm waiting for the crash as I write to you.

Update: . It crashed. I removed gpgsm, then it crashed again.

So the problem cannot be in systemd-sysv and cannot be in gpgsm. It must be somewhere else. So what else might I have done to trigger the crash?

While thinking about the issue, I looked at my ~/.muttrc file and noticed this old line:

Code: Select all

set editor="qemacs"

The qemacs-nox package was removed from Debian ages ago, but I still had it installed on my system and mutt was using it to send emails. I played with it and the computer crashed.

So I removed qemacs-nox and now I'm waiting for another crash. I'll keep you posted.

In the meantime, thank you for your help.

Sincerely,
- Soul
.

[Head_on_a_Stick]

If you are shutting down the computer after these freezes then try enabling persistent logging:

Code: Select all

# mkdir -p /var/log/journal # delete the directory to return to non-persistent logging

Then use the systemd journal to check the last boot:

Code: Select all

journalctl -r -b -1

The coredumpctl command can also be very useful in situations like this.

[Soul Singin']

try enabling persistent logging:

There's a good idea. Thank you! I just enabled it.

And in general, thank you for your help! .

This time, I think I've identified the real issue and I think it was qemacs-nox. My system has been crash free since I removed that package.

I have not tried sending any emails yet, but I will. First, I want to test this configuration a little bit. If it runs smoothly, then I will try sending some emails. And if the system is still crash-free after that, then we should be able to confidently say that the problem was qemacs-nox.

Keep your fingers crossed for me!

Thanks,
- Soul
.

[Soul Singin']

This time, I think I've identified the real issue

I was wrong. It crashed again.

Here is the relevant section from the journal log:

Code: Select all

-- Logs begin at Fri 2019-05-17 10:02:26 EDT, end at Fri 2019-05-17 15:43:19 EDT. --
May 17 15:37:20 inspiron kernel: perf: interrupt took too long (5197 > 4963), lowering kernel.perf_event_max_sample_rate to 38250

What does that mean?
.

[Wheelerof4te]

^Nothing:
https://bbs.archlinux.org/viewtopic.php?id=187636

This is informational and nothing to worry about. It has to do with the Linux perf tool which is included in the kernel. The kernel automagically determines the sample rate that could be used without impacting system performance too much; and it logs this even when perf isn't active, or even installed. Messages like this are triggered by high(er) system load or a cpu that is scaling

Or that you CPU is failing or scaling wrong.
You should try a live distro and see if it crashes. If it doesn't, maybe consider reinstalling Debian.

[Head_on_a_Stick]

The journal can filter according to message priority:

-p, --priority=
Filter output by message priorities or priority ranges. Takes either a single numeric or textual log level (i.e. between 0/"emerg" and 7/"debug"), or a range of numeric/text log levels in the form FROM..TO. The log levels are the usual syslog log levels as documented in syslog(3), i.e. "emerg" (0), "alert" (1), "crit" (2), "err" (3), "warning" (4), "notice" (5), "info" (6), "debug" (7). If a single log level is specified, all messages with this log level or a lower (hence more important) log level are shown. If a range is specified, all messages within the range are shown, including both the start and the end value of the range. This will add "PRIORITY=" matches for the specified priorities.

So perhaps try

Code: Select all

journalctl -b -1 -p 3

But I think this is beginning to sound like a hardware problem so check the memory & drive health.

[Soul Singin']

It just crashed again.

Following your suggestions, I found something interesting in the logs:

Code: Select all

$ journalctl  -b -1 -p4
-- Logs begin at Fri 2019-05-17 10:02:26 EDT, end at Fri 2019-05-17 16:47:13 EDT. --
May 17 15:34:54 inspiron kernel: core: CPUID marked event: 'bus cycles' unavailable
May 17 15:34:54 inspiron kernel: mtrr: your CPUs had inconsistent variable MTRR settings

What does that mean?


EDIT: . it just crashed again. I switched to the previous kernel (4.9.0-7). Let's see if it happens here.

In the meantime, I found this (and a similar message from the previous boot):

Code: Select all

$ journalctl -r -b -0 -p4
-- Logs begin at Fri 2019-05-17 10:02:26 EDT, end at Fri 2019-05-17 17:09:08 EDT. --
May 17 17:04:39 inspiron kernel: [drm:ironlake_irq_handler [i915]] *ERROR* PCH transcoder A FIFO underrun
May 17 17:04:39 inspiron kernel: [drm:ironlake_irq_handler [i915]] *ERROR* CPU pipe A FIFO underrun

EDIT 2: . And another crash. I'm now on the new kernel (4.9.0-9), released a few days ago. I see the same messages in the new log. Let's what happens now.
.

[Soul Singin']

I'm now on the new kernel (4.9.0-9), released a few days ago.

So far all is good with the new kernel. No crashes.
.

[sickpig]

ok great! happy the issue is fixed, hope it stays that way

[Soul Singin']

ok great! happy the issue is fixed, hope it stays that way

Thanks for the good wishes, but it crashed again.

When I rebooted, I noticed that there were orphaned inodes associated with UID=109 and GID=119, which corresponds to the MySQL user on my system. Could the issue have been occurring there?

We'll find out. I purged all of the MariaDB packages. Let's see what happens now.


EDIT: . Crashed again.
.

[Head_on_a_Stick]

*cough*

I think this is beginning to sound like a hardware problem so check the memory & drive health

[Soul Singin']

I think this is beginning to sound like a hardware problem so check the memory & drive health

LoL! You may be right.

But before I go out and purchase a new machine, I want to be sure that this not a software issue. So I did what any good newb would do and ran:

Code: Select all

# apt-get install --reinstall debian

That didn't work very well, so I downloaded a Buster installation DVD and I am now (at this moment) installing over the old installation.

If it is a software issue, then maybe clearing out 10 years of software configurations will fix it. (And I will have an upgraded system too).

Goodbye Stretch! It's been nice.
Hope you find your paradise!
Tried to see your point of view,
Hope your dreams will all come true.

Goodbye Lenny! Goodbye Squeeze!
Will we ever meet again?
Feel no sorrow, feel no shame,
Come tomorrow, feel no pain.

.

[Soul Singin']

I have said this before, but ... My machine seems to work fine now. I do not see any "FIFO underruns" in the output of journalctl. Please keep your fingers crossed for me.

Thank you to: Head_on_a_Stick, sickpig, GarryRicketson, golinux, debiman and Wheelerof4te for all of the help and support that you gave me. You all have been wonderful. Thank you! .

Sincerely,
- Soul
.