Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

A question about system logs

If none of the specific sub-forums seem right for your thread, ask here.
Post Reply
Message
Author
Marie SWE
Posts: 241
Joined: 2021-04-06 22:14
Location: Sweden / Linköping
Has thanked: 7 times
Been thanked: 9 times

A question about system logs

#1 Post by Marie SWE »

Hi :)

Today was the second time that my system crash/hang when the computer should startup from sleep mode.. (within the last 1½ months)
Are there any good programs to browse through all the system logs in a easy way, so I do not have to open one text file at a time?

Edit:
Debian Buster

Edit 2:

I did change subject instead of starting a new thread. 8)
my system has crashed when the computer should startup from sleep mode.. I need to find out what is the problem. :thinking:
Last edited by Marie SWE on 2021-09-09 23:56, edited 1 time in total.
Why make things complicated in life, if you can make it easier for yourself... Do it. ;o)
You only have one life, so make the most of it and enjoy it while you can.

Marie SWE
Posts: 241
Joined: 2021-04-06 22:14
Location: Sweden / Linköping
Has thanked: 7 times
Been thanked: 9 times

Re: A question about which program for logs

#2 Post by Marie SWE »

I found three viewing programs in synaptic packet manager.. :)

No time or date stamps in boot.log :shock:
Only fail I found was this
[[0;1;31mFAILED[0m] Failed to start [0;1;39mRaise network interfaces[0m.
I guess this is my inactive onboard networkcard

which more logs have to do with boot, so I can see why my computer did crash from sleep startup
Why make things complicated in life, if you can make it easier for yourself... Do it. ;o)
You only have one life, so make the most of it and enjoy it while you can.

Aki
Global Moderator
Global Moderator
Posts: 2816
Joined: 2014-07-20 18:12
Location: Europe
Has thanked: 68 times
Been thanked: 382 times

Re: A question about system logs

#3 Post by Aki »

Hello Marie SWE,

If I read correctly, you are searching some hints to understand how to detect possible causes of system crash/hang after suspend/resume. This topic is a complex tasks and the analysis is more difficult if the issue is not systematically reproducible.

Of course, system logs are the first place to search for clues: some hints can be found here [1][2].

Try to narrow down what subsystem (kernel ? GPU kernel module ? hardware failure ? other ?) could be involved according to visible symptoms and try to obtain more debug information from the involved subsystems. Be aware that the operating system could be unable to log the last system messages if the linux kernel completely hangs.

In the log file, the log entry about the start of suspend activities is usually the starting point to analyze system messages, for example:

Code: Select all

lug 23 01:54:19 debian systemd[1]: Starting Suspend...
[1] https://01.org/linuxgraphics/documentat ... ume-issues
[2] https://01.org/blogs/rzhang/2015/best-p ... ate-issues
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄⠀

Marie SWE
Posts: 241
Joined: 2021-04-06 22:14
Location: Sweden / Linköping
Has thanked: 7 times
Been thanked: 9 times

Re: A question about system logs

#4 Post by Marie SWE »

Aki wrote: 2021-09-10 08:40 Hello Marie SWE,

If I read correctly, you are searching some hints to understand how to detect possible causes of system crash/hang after suspend/resume. This topic is a complex tasks and the analysis is more difficult if the issue is not systematically reproducible.

Of course, system logs are the first place to search for clues: some hints can be found here [1][2].

Try to narrow down what subsystem (kernel ? GPU kernel module ? hardware failure ? other ?) could be involved according to visible symptoms and try to obtain more debug information from the involved subsystems. Be aware that the operating system could be unable to log the last system messages if the linux kernel completely hangs.

In the log file, the log entry about the start of suspend activities is usually the starting point to analyze system messages, for example:

Code: Select all

lug 23 01:54:19 debian systemd[1]: Starting Suspend...
[1] https://01.org/linuxgraphics/documentat ... ume-issues
[2] https://01.org/blogs/rzhang/2015/best-p ... ate-issues
Hi Aki :D

Yes, you read it correctly.. I know how to do all this in windows.... but unfortunately for me, Linux isn't windows. :lol:
So I must to learn how to do this in linux..
a saying: once is never, twice is a coincidence, a third is a problem/habit
and this crash from sleep/hibernation(ram to disk) has happen twice.
What is the correct word for setting system ram to disk? is it "sleep mode" or is it hibernation? :thinking:
So I want to be prepared for the third time.

To answer your questions, hardware, kernel and so on.

Code: Select all

user@Big-i3:~$ inxi -Fxz
System:
  Host: Big-i3 Kernel: 4.19.0-14-amd64 x86_64 bits: 64 compiler: gcc 
  v: 8.3.0 Desktop: Xfce 4.12.4 Distro: Debian GNU/Linux 10 (buster) 
Machine:
  Type: Desktop Mobo: Intel model: H310 serial: <filter> 
  UEFI [Legacy]: American Megatrends v: 5.12 date: 04/09/2019 
CPU:
  Topology: Quad Core model: Intel Core i3-8100 bits: 64 type: MCP 
  arch: Kaby Lake rev: B L2 cache: 6144 KiB 
  flags: lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx bogomips: 28800 
  Speed: 1100 MHz min/max: 800/3600 MHz Core speeds (MHz): 1: 1045 2: 1021 
  3: 1034 4: 1056 
Graphics:
  Device-1: Intel 8th Gen Core Processor Gaussian Mixture Model driver: i915 
  v: kernel bus ID: 00:02.0 
  Display: x11 server: X.Org 1.20.4 driver: modesetting unloaded: fbdev,vesa 
  resolution: 1440x900~60Hz, 1280x720~60Hz 
  OpenGL: renderer: Mesa DRI Intel UHD Graphics 630 (Coffeelake 3x8 GT2) 
  v: 4.5 Mesa 18.3.6 direct render: Yes 
Audio:
  Device-1: Intel 200 Series PCH HD Audio driver: snd_hda_intel v: kernel 
  bus ID: 00:1f.3 
  Sound Server: ALSA v: k4.19.0-14-amd64 
Network:
  Device-1: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet 
  driver: r8169 v: kernel port: e000 bus ID: 01:00.0 
  IF: enp1s0 state: up speed: 1000 Mbps duplex: full mac: <filter> 
  IF-ID-1: tun0 state: unknown speed: 10 Mbps duplex: full mac: N/A 
Drives:
  Local Storage: total: 1.20 TiB used: 39.37 GiB (3.2%) 
  ID-1: /dev/sda vendor: Western Digital model: WD3200AAJS-60M0A0 
  size: 298.09 GiB 
  ID-2: /dev/sdb vendor: Hitachi model: HDS721010CLA332 size: 931.51 GiB 
Partition:
  ID-1: / size: 121.60 GiB used: 37.64 GiB (31.0%) fs: ext4 dev: /dev/sda1 
  ID-2: swap-1 size: 25.00 GiB used: 1.73 GiB (6.9%) fs: swap dev: /dev/sda3 
Sensors:
  System Temperatures: cpu: 36.0 C mobo: N/A 
  Fan Speeds (RPM): N/A 
Info:
  Processes: 214 Uptime: 23h 09m Memory: 7.71 GiB used: 3.92 GiB (50.9%) 
  Init: systemd runlevel: 5 Compilers: gcc: 8.3.0 Shell: bash v: 5.0.3 
  inxi: 3.0.32 
user@Big-i3:~$ 
I almost dare say that it isn't a hardware failure, But I can run some test programs for RAM, HDD, etc. to backup that statement/guessing :)

Is it possible to see all installed programs and what date and time they was installed?

I'm thinking if a newly installed program can be the cause. I did install some program(debian repos only) But i don't remember if that was before the first crash or after the first crash. :thinking:
I haven't installed any updates the last four months.. I have not change any hardware since the install of Debian in April..... no, I did add a new network card PCIe.. but when.. perhaps three months ago.. I'm not entirely sure :oops: I did disabled onboard network in Bios as it only wanted to go in 100Mbit mode..
So my guess from a windows perspective it can probably be new software that is the most likely reason.
But it could just as well be that the system has been damaged for an unclear reason.

Now it's time for some more coffee, so I wake up some more. :mrgreen: :mrgreen: :mrgreen:

EDIT:
I forgot to describe the startup :oops:
Power on --> Bios startscreen --> Grub dualboot menu --> Debian autostart 15seconds --> Debian boot picture ---> Then black as it normaly does before autologin.. but here it hangs with black screens.. nothing works.. Num lock, Caps lock nothing responds... only thing to do is a reboot with power button off (Power settings when Powerbutton pressed =shutdown)
Why make things complicated in life, if you can make it easier for yourself... Do it. ;o)
You only have one life, so make the most of it and enjoy it while you can.

Victor Brand
Posts: 26
Joined: 2021-09-10 16:35
Location: St. Petersburg, Russia
Been thanked: 4 times

Re: A question about system logs

#5 Post by Victor Brand »

All system-wide logs in Linux are usually saved under /var/log directory.
Marie SWE wrote: 2021-09-10 14:10 Is it possible to see all installed programs and what date and time they was installed?
Yes, check /var/log/apt/history.log.

But this will hardly help you in your problem. It is unlikely connected with installing some desktop applications. Hibernating and resuming from hibernation involves the kernel and initramfs, and also systemd.

By the way, hibernating in linux is a bit tricky thing, and there are reports on similar bug in Debian and its workaround, please check:
https://www.reddit.com/r/debian/comment ... bernation/

And also this topic may be helpful:
https://wiki.archlinux.org/title/Power_ ... nsistently
(especially the note on buggy watchdog settings)

There could be some other issues like some services don't run properly upon resume (e.g. bluetooth or screensaver or pulseaudio), but I've never stumbled upon these, mostly because I don't use hibernation :)

Marie SWE
Posts: 241
Joined: 2021-04-06 22:14
Location: Sweden / Linköping
Has thanked: 7 times
Been thanked: 9 times

Re: A question about system logs

#6 Post by Marie SWE »

Victor Brand wrote: 2021-09-10 16:52 All system-wide logs in Linux are usually saved under /var/log directory.
Marie SWE wrote: 2021-09-10 14:10 Is it possible to see all installed programs and what date and time they was installed?
Yes, check /var/log/apt/history.log.

But this will hardly help you in your problem. It is unlikely connected with installing some desktop applications. Hibernating and resuming from hibernation involves the kernel and initramfs, and also systemd.

By the way, hibernating in linux is a bit tricky thing, and there are reports on similar bug in Debian and its workaround, please check:
https://www.reddit.com/r/debian/comment ... bernation/

And also this topic may be helpful:
https://wiki.archlinux.org/title/Power_ ... nsistently
(especially the note on buggy watchdog settings)

There could be some other issues like some services don't run properly upon resume (e.g. bluetooth or screensaver or pulseaudio), but I've never stumbled upon these, mostly because I don't use hibernation :)
Thanks for the tip :D
Unfortunately the apt history log didn't say anything. According to it, I have not installed anything since 10th of August.. and I know I installed a video editor a week ago.
:oops: .oO(whispering)... I never use the terminal unless I'm forced to, I don't like the terminal's environment, it feels so 90s... but do not tell anyone :wink:

I am reading through all my logs now... but I have some more to read before I can say anything more.
I did found out that the first crash was on August 22nd.. so some progress
So I do not think it is a bug, because the hibernation has worked from my installation in April 2021 until the 22nd when it happened for the first time. So I need to figure out what I did in July/August when it takes around two and a half weeks between the crashes.

I will read all the links a bit more after I get me some dinner. :P
Why make things complicated in life, if you can make it easier for yourself... Do it. ;o)
You only have one life, so make the most of it and enjoy it while you can.

Victor Brand
Posts: 26
Joined: 2021-09-10 16:35
Location: St. Petersburg, Russia
Been thanked: 4 times

Re: A question about system logs

#7 Post by Victor Brand »

Marie SWE wrote: 2021-09-10 17:56So I need to figure out what I did in July/August when it takes around two and a half weeks between the crashes.
It's an unfortunate glitch because it seems to be hard to consistently reproduce it. First of all, try to figure out whether there were kernel upgrades during that period.

Then, looking through the logs is definitely better to perform via terminal. You don't need to read through all the nonsense listed there, just search for the things you are interested in. It's better to check not only boot.log, but also kern.log and syslog (if these are present) for error messages in the period before the hibernation (because incorrect resume might be caused by errors during the "falling asleep").

If you ever have this issue again, run in the terminal after you boot into your system:

Code: Select all

sudo cat /var/log/kern.log | grep -iE '(error|failed)'
and then:

Code: Select all

sudo cat /var/log/syslog | grep -iE '(error|failed)'
and see what these commands print. (I don't run those logging daemons and therefore don't have these files on my system, so I can't check whether or not you need root privileges in order to access them; but IIRC it's necessary, that's why there is 'sudo' in the beginning).

Marie SWE
Posts: 241
Joined: 2021-04-06 22:14
Location: Sweden / Linköping
Has thanked: 7 times
Been thanked: 9 times

Re: A question about system logs

#8 Post by Marie SWE »

Victor Brand wrote: 2021-09-10 20:04
Marie SWE wrote: 2021-09-10 17:56So I need to figure out what I did in July/August when it takes around two and a half weeks between the crashes.
It's an unfortunate glitch because it seems to be hard to consistently reproduce it. First of all, try to figure out whether there were kernel upgrades during that period.

Then, looking through the logs is definitely better to perform via terminal. You don't need to read through all the nonsense listed there, just search for the things you are interested in. It's better to check not only boot.log, but also kern.log and syslog (if these are present) for error messages in the period before the hibernation (because incorrect resume might be caused by errors during the "falling asleep").

If you ever have this issue again, run in the terminal after you boot into your system:

Code: Select all

sudo cat /var/log/kern.log | grep -iE '(error|failed)'
and then:

Code: Select all

sudo cat /var/log/syslog | grep -iE '(error|failed)'
and see what these commands print. (I don't run those logging daemons and therefore don't have these files on my system, so I can't check whether or not you need root privileges in order to access them; but IIRC it's necessary, that's why there is 'sudo' in the beginning).
I'm still looking in the logs.... But I want to respond a little quickly to your post. :)
EDIT: You asked about: Quote: First of all, try to figure out whether there were kernel upgrades during that period. end quote
I still have Kernel: 4.19.0-14-amd64 as showed in inxi output.. so no upgrade during that period.
I have not upgrade or updated my system the last 3 or 4 or 5months. 177updates are awaiting. :oops:
Is it possible to see when updates was last installed
and i still have no clue how to see when i installed programs date/time for install
end edit

When it comes to Boot log, it is useless in my opinion..
There are no timestamps in it so I can't compare to other logs, I know when I restart the computer from hibernation at.15:20:23 and I know when I restarted the computer after the hang 16:15:12
If I try to open boot log in Pluma, it looks like this. Sorry for the swedish if you want i can translate it, but i guess it's a standard error message
((Ignore the location, i have copied the logs to another disk so they remain untouched))
2021-09-10_23-17-04.png
2021-09-10_23-17-04.png (51.23 KiB) Viewed 8013 times
.
If I open boot log in the log program then it looks like this...... no date and time stamps.
2021-09-10_23-19-29.png
2021-09-10_23-19-29.png (114.25 KiB) Viewed 8012 times
Why make things complicated in life, if you can make it easier for yourself... Do it. ;o)
You only have one life, so make the most of it and enjoy it while you can.

Victor Brand
Posts: 26
Joined: 2021-09-10 16:35
Location: St. Petersburg, Russia
Been thanked: 4 times

Re: A question about system logs

#9 Post by Victor Brand »

Marie SWE wrote: 2021-09-10 21:36 Sorry for the swedish if you want i can translate it, but i guess it's a standard error message
I can understand the general sense of the message. I don't speak Swedish, but I read in German btw. The problem is, you need root privileges in order to access that file.

But frankly saying folks usually don't use pluma to read logs :) They use the command line for that. Don't worry about your old habits, just start using CLI, and with certain practice you'll understand how convenient and powerful it is. The reason is, the logs are usually bloated with various stuff like diagnostic messages etc. For example, only during the boot the kernel alone produces more than 1000 lines of messages. Thus it's useless to read the logs like, say, George Martin's books. This way, you'll waste much time and highly likely won't find anything useful. The solution is to filter the logs in search for the desired clues. In CLI, there are commands called filters which (suddenly) filter the input according to certain patterns. The most frequently used one of them is, probably, grep, which takes the input and prints only the strings matching the specified pattern. E.g. the command

Code: Select all

sudo cat /var/kern.log | grep i915
will show kernel messages related to the intel i915 video driver. You don't need to scroll through all the log in search of them and you are sure you don't miss anything.

Moreover, there are logs which are not stored in text files. Those are managed by the journald daemon and you can access them only through the console command journalctl. Kernel messages are accessed through the command dmesg (must be run with sudo). For example, I disabled almost all logging (I don't need it), so I don't have kern.log on my computer, but I can always read what the kernel says via 'sudo dmesg' (ofc, these messages do not survive reboot, but that's unnecessary for me; kern.log, OTOH, is persistent and you can read it after reboot or shutdown; kern.log contains messages produced by the dmesg command).

The commands I provided you earlier display system messages related to errors and failures, which can be useful in order to determine whether something goes wrong in your system. Ofc, you can scroll through the log files in search for such messages, but that's unreliable and time consuming.

boot.log is not completely 'useless', and it keeps timestamps (you see that 0:32 etc). But it contains just diagnostic messages your system prints on startup. It can be useful if, for example, you see some error messages during the boot, but they vanish so quickly that you cannot read them. Not a problem, they are stored in boot.log, and you can access them later and study thoroughly. (Error and warning messages displayed during the boot can usually be accessed via dmesg as well).

Regarding your problem, I can see according to the boot.log that you probably have some encrypted volumes. Are your disks encrypted? Is your swap partition encrypted as well? This directly affects hibernation (suspend-to-disk) and makes it a bit more complicated. (Apparently if your root and home partitions are encrypted, your swap, if it exists, must be encrypted as well, otherwise unencrypted data may be stored to swap and your security will be compromised).

Aki
Global Moderator
Global Moderator
Posts: 2816
Joined: 2014-07-20 18:12
Location: Europe
Has thanked: 68 times
Been thanked: 382 times

Re: A question about system logs

#10 Post by Aki »

Hello Marie SWE,

I'll try to answer to some of the unanswered questions.
Marie SWE wrote: 2021-09-09 23:05 No time or date stamps in boot.log :shock: Only fail I found was this

Code: Select all

[[0;1;31mFAILED[0m] Failed to start [0;1;39mRaise network interfaces[0m.
I guess this is my inactive onboard networkcard
The /var/log/boot.log in not a plain text file, but it contains "escape characters": see here [1] to see how to manage them.
Marie SWE wrote: 2021-09-09 23:05 which more logs have to do with boot, so I can see why my computer did crash from sleep startup
I saw you already received advice about it. Here [2] you can find the list of the log files the system keeps: /var/log/syslog usually receives most of the messages from kernel and services. In addition, with the introduction of systemd, the systemd journal (journalct command, see [3]) is often the first collector of the majority of system log messages: it's not available if a different init system is used, of course. Furthermore, according the init system used (e.g, systemd versus sysvinit [3.1]) different boot sequences (parallel versus sequential start of system services) must be taken into account during analysis.
Marie SWE wrote: 2021-09-09 19:49 What is the correct word for setting system ram to disk? is it "sleep mode" or is it hibernation? :thinking:
There are usually two types of computer suspension: Suspending to Ram (STR) (alias supend) and Supend To Disk (STD) (alias hibernation) [4]
Marie SWE wrote: 2021-09-10 14:10 I forgot to describe the startup :oops:
Power on --> Bios startscreen --> Grub dualboot menu --> Debian autostart 15seconds --> Debian boot picture ---> Then black as it normaly does before autologin.. but here it hangs with black screens.. nothing works.. Num lock, Caps lock nothing responds... only thing to do is a reboot with power button off (Power settings when Powerbutton pressed =shutdown)
It could be useful to understand at which stage of the boot process the system hangs. You probably have to increase the "verbosity" of the kernel (remove the kernel quiet parameter, add the debug parameter, add other parameters to increase the verbosity of the early stages of boot). Furthermore, you could disinstall plymouth package to have a full textual boot to watch kernel and system logs in real time during boot after a suspension (STR) or hibernation (STD). A serial interface could be very useful, but it’s rare nowdays.

Last but not least, please send log messages as text file (in the message body between code tags or attached as compressed file) rather then linking pictures to messages.

Hope that helps.

[1] https://wiki.debian.org/bootlogd
[2] https://debian-handbook.info/browse/sta ... yslog.html
[3] https://manpages.debian.org/bullseye/sy ... .1.en.html
[3.1] https://wiki.debian.org/Init
[4] https://www.kernel.org/doc/Documentatio ... states.txt
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄⠀

Marie SWE
Posts: 241
Joined: 2021-04-06 22:14
Location: Sweden / Linköping
Has thanked: 7 times
Been thanked: 9 times

Re: A question about system logs

#11 Post by Marie SWE »

Hello again :D
Sorry for my two or three months of absence. :oops: :oops: This fall became a bit chaotic for me, so all computer projects ended up on ice for a while.
Now it's 16 degrees Celsius below zero here, so it's not fun to do things outdoors :roll: so now it's time to get on with my computer projects again.

Regarding this thread.
The phenomenon has not occurred again, I did change from my HDD swap partition to a SSD swap partition. :roll: :oops: :|
The story..... One morning, about three weeks ago when I had my first cup of coffee in my hand and was newly awake.. and it took 26 minutes to start the damn computer from hibernation do to swapdeath or I/O lockup you name it..... and I just got soooo pissed off that day.. the computer was literally just 4minutes from going out the window, so damn pissedoff I was that day, ohh my god.. :oops: I opened the damn computer while it was powered on and fully booted, I just tore the cables from my dvd-rw in anger and took an old 128GB ssd I had lying around and just connected it in hotswap style and then set it up for my new swap partition.. no more 20+ minutes. :mrgreen:
Note, for those who read this, you should definitely not plug a new hard drive into a desktop computer in this way while it is on. that is not recommended.
My system and home is still on the same HDD and it's working fine so far. But since I don't have any long uptime yet, I don't dare to say that it is solved yet... so to be continued. 8)
Why make things complicated in life, if you can make it easier for yourself... Do it. ;o)
You only have one life, so make the most of it and enjoy it while you can.

clementishutin
Posts: 37
Joined: 2021-12-10 11:59
Been thanked: 3 times

Re: A question about system logs

#12 Post by clementishutin »

Try to figure out which subsystem (kernel? GPU kernel module?) is at fault.
According to apparent symptoms, hardware failure? other?) could be involved, and more debug information from the associated subsystems should be sought.
If the linux kernel totally hangs, the operating system may be unable to log the most recent system notifications.

Post Reply