Qnap NAS TS-259 pro+ hangs every now and then when booting Debian 12

[quack]

Hello everybody,

I'm new to Debian, but not new to computers. I'm also to some extent familiar with Linux and I'm familiar with the command line / shell.

I have an old Qnap NAS "TS-259 pro+" which is out of support now nice several years. Therefore I decided to give it a try and install Debian on it. The NAS is still x32-bit, so I installed the 32-bit edition of Debian.
I took the DVD image from here https://cdimage.debian.org/debian-cd/cu ... 6/iso-dvd/ and copied it with Rufus to an USB stick. Installation went fine. No obvious problems.

As boot device I use an USB stick with 8GB, because I want to be able to revert to the original firmware, in case things do not work out as expected. (stick is USB 3, while USB port of NAS is USB 2 - just in case this does matter).


The Debian installation on my NAS basically works fine and is working as expected.

BUT every now and then (about 4-6 times out of 10) I have problems booting the NAS. It either boots fine or it hangs after "initial ramdisk ...".

After loading the ramdisk the screen goes black with just a blinking cursor in the upper left corner.
I know that this is expected behavior (for several seconds) - but it's also expected to show the login screen after maybe half a minute.

And here the NAS get's stuck.

I tried removing the "quiet" parameter from GRUB, but that didn't help.
When it boots, I get all the expected messages. If it doesn't boot, I don't get any of those messages. Not a single line...

I changed the kernel params in Grub to look like this:

Code: Select all

"debug earlycon ignore-loglevel keep-bootcon"

hoping to get more information.

This has not led to any significant change.
I get no log output when the boot process hangs.
(Of corse I get plenty of output when the boot process is successful)
So no additional information where it might get stuck.


Oh and just to anticipate some obvious questions:

I have checked the HW of the USB stick (at least 2x). It is fine.
I used a different stick (just in case) --> same behavior.
I tried different USB ports of the NAS --> same behavior.

Some more information, that I thought might help you:

> uname -a
Linux debian-nas 6.1.0-18-686-pae #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) i686 GNU/Linux


Any ideas?


Many thanks, quack

[quack]

I tried a few more things and also installed Debian on one partition of the spinning disks.

This is not the way the system is designed, because I can't boot automatically from that disk.
Instead I must press F11 during the boot sequence, then the BIOS displays a selection dialogue and there I can not only select the USB devices, but also the spinning disks.
But for testing purposes this is ok for the moment.

Doing so I found out the following:

• Also installation of Debian from the stick is affected from this behavior. I just didn't notice the first time.

• Directly after the installation and trying to boot the very first time, I had the impression, it won't work at all, since it took about 6-8 repetitions until I was able to successfully boot into Debian.

• Then I had about as many reboots, that worked completely successful.

• It might be, that reboots (issued with the "reboot" command on command line) have a much better probability to be successful than "cold" reboots (shutting down the NAS using the command "shutdown") and waiting a few minutes until the next retry). But this is something I must still to some repetitions, to be sure about it.

Now I am in about the same situation an before. About 4-6 out of 10 attempts hang after the "initial ramdisk..."

To me it seems, that the behavior is not related to booting from USB stick, but might be some timing issue.

Can maybe someone explain to me, what exactly happens after "initial ramdisk.." is displayed?
(pointing to some location with the information is just fine)

Does anybody know some commands or switches, that might change timing during the boot sequence?
Or does anybody know some commands or switches, that might give more logging output directly after "initial ramdisk..."

Any help would really been highly appreciated.

Thanks, quack

[Aki]

Hello,

Can maybe someone explain to me, what exactly happens after "initial ramdisk.." is displayed?

The kernel boots the operating system from the initramfs (as "root" disk) then it changes the root file system to the "final" root disk partition.

Or does anybody know some commands or switches, that might give more logging output directly after "initial ramdisk..."

You may:

• check/share system logs searching for clues
• start (from grub) the operating system (Debian) in recovery mode (to check if the boot success/failure ratio is the same)
• replace the "quiet" with the "debug" kernel parameter (using grub)
• also add the "nomodeset" kernel parameter together with the "debug"

According to [1], your NAS has with 1 GB RAM. It may be worth having a look at Debian 12 memory and disk space requirements and what can be done about it [2].

The Debian Wiki gives some advices about initramfs debugging [3]

Hope this helps. Please let me know.

---
[1] https://www.qnap.com/it-it/product/ts-259%20pro+/specs/hardware
[2] Debian GNU/Linux Installation Guide - Chapter 2. System Requirements > 2.5. Memory and Disk Space Requirements
[3] Debian Wiki - InitramfsDebug

[quack]

Thanks for your answer. It really IS appreciated.

• check/share system logs searching for clues

Is this what I get with dmesg? If so, how do I make sure, I select the entries from an UNsuccessful boot?

• start (from grub) the operating system (Debian) in recovery mode (to check if the boot success/failure ratio is the same)

Will do this, later this day or tomorrow and report.

• replace the "quiet" with the "debug" kernel parameter (using grub)
• also add the "nomodeset" kernel parameter together with the "debug"

I did this in the past already. behavior didn't change.
Meaning, if the boot process hangs after "initial ramdisk..." I still don't get any output at all (on the screen).

According to [1], your NAS has with 1 GB RAM. It may be worth having a look at Debian 12 memory and disk space requirements and what can be done about it [2].

I did look at it, and I'm confident, I'm on the save side, since I have a "No Desktop" installation which according to the memory requirements needs 256MB minimal RAM and 512MG recommended RAM. Having 1GB RAM should be fine though.

The Debian Wiki gives some advices about initramfs debugging [3]

This looks very interesting. Thanks for that link.
I will look deeper into it later today or tomorrow and report the results.

Thanks, quack

[quack]

So I did go through all the suggestions.

Using the Recovery Mode didn't change it.
I used it booting from spinning disks as well as booting from USB stick.
Either it boots fine or it hangs directly after "initial ramdisk...".
Success ratio did not change.

Looking at the wiki and man pages on initramfs I learned, that debug=vc writes the debug output to screen, so I did this. I also used break=something (with almost every possible value here).
Result is still the same. It either boots or it hangs (without any debug output) directly after "initial ramdisk...".

To me it looks like the machine isn't even able to start / mount the initramfs.
Does this make any sense?

And if it does, what could be a possible solution?

[Aki]

What about the rootdelay option ? Give it a lot of time

[quack]

Oh, I forgot to mention this one.
Yes, I also tried rootdelay=9 with the same (i.e. no) effect.
It either boots fine or it hangs without any output after the ramdisk.

[Aki]

Hello,

To me it looks like the machine isn't even able to start / mount the initramfs.

It might be useful to take a look at a video recording (e.g. using your smartphone) of the computer's console when the computer fails to boot with the following kernel parameters (removing "quiet" and "splash" kernel parameters, if any):

Code: Select all

debug nomodeset break=premount

[quack]

I did record two videos. One from a successful boot and one from a failed boot, so you can see the difference. The last seconds of the failed boot, where you can see the lank screen with just the blinking cursor in the upper left corner, is where it hangs (if it hangs).
If it boots successfully, then this screen is only seen for a fraction of a second and immediately flooded with logging (as you can see in the other video).

I'm sorry about the quality of the video. Obviously the camera has a hard time to focus on a computer monitor. But I think you will get a fairly good idea of the boot process.
The videos are about 130MB / 160MB each.

failed boot
sucessful boot

[Aki]

Hello,

Videos unfortunately cannot be downloaded.

If it boots successfully, then this screen is only seen for a fraction of a second and immediately flooded with logging (as you can see in the other video).

If you passed the kernel parameters I asked for in my previous post (debug break=premount), it should stop with a "(initramfs)" prompt if it succeeds in booting. Since this is not happening (according to your quote above), something is not adding up.

[quack]

I changed the security settings. Download should now work.

Sure I did pass the kernel params. And they work as expected.
If it boots successfully, it stops at (initramfs).
You will see this in the video with the successful boot.
If it comes that far, it will boot and work as expected.

If it hangs, it hangs before that.
That's the reason, why I am suspecting, that the machine isn't even able to start / mount the initramfs.

[Aki]

Hello,

If it hangs, it hangs before that. That's the reason, why I am suspecting, that the machine isn't even able to start / mount the initramfs.

grub loads both the kernel and the initramfs, but I suspect the kernel is hanging in the early stages of startup.

You could try to get more clues by using the following additional parameters to the kernel with grub (to try to debug the early stages of kernel startup) removing the "quiet" kernel parameter:

Code: Select all

earlyprintk=vga printk.time=1 initcall_debug

What is the device you are booting from in the videos ?

Hope this helps.

[quack]

but I suspect the kernel is hanging in the early stages of startup.

Well, must the the very early stages...
The params you supplied didn't change anything.
If it hangs, I still see no line with logging information.

What is the device you are booting from

The machine is my NAS (Qnap TS-259 Pro+).
The device where the kernel and the OS is stored, is an external USB stick.

[Aki]

Based on the information given so far, I would suspect a hardware, computer, or boot media-related cause.

[quack]

I would usually too. But booting from the original firmware, I don't have any issue at all.
That's what driving me nuts. I know and see, that it works with the original firmware....

The only real difference, when booting the firmware is, that the firmware is installed in some onboard flash memory. To the BIOS / OS that flash memory behaves as just some USB device.

I haven't yet tried to use that internal flash memory, since I wanted to be able to go back to the original firmware, if things don't work out.

The other thing is, that the flash memory only has 512MB. That's not enough to install the whole Debian. Do you know any trick, what I could do during installation of Debian to use that 512MB of internal flash? Maybe just putting the kernel there and having the rest of Debian still on the external USB stick.
(Just in case, I decide to go in that direction even if I don't know, if it will work or not)

[Aki]

Hello,

The only real difference, when booting the firmware is, that the firmware is installed in some onboard flash memory. To the BIOS / OS that flash memory behaves as just some USB device.

The BIOS on your NAS is very old (2008).

I would not be so sure that booting from a USB 3.0 pen drive (like yours) is the same as booting from the onboard flash memory.

Instead, you can try booting Debian from an external USB 2.0 enclosure with a mechanical hard drive (instead of the pen drive), if you have one to install Debian on.

I haven't yet tried to use that internal flash memory, since I wanted to be able to go back to the original firmware, if things don't work out.

I agree with you: leave the original firmware on the internal flash memory.

[quack]

Instead, you can try booting Debian from an external USB 2.0 enclosure with a mechanical hard drive

During my experiments with my NAS I also installed Debian on the local Hard Disk and booted from there. This should be completely fine for the BIOS or HW, but I had the same issue, that 4 out of 10 boots were unsuccessful.
I will see, if I find some USB2 device to test it with, but to be honest, I doubt it.

I found this in another thread in a Qnap forum:

Installation [of Debian] on x86-based NAS is almost identical to installation on a standard PC. Boot your Debian installer from USB, [...] put "/boot" on the DOM, and everything else into your [HD].

If I understand this correctly, assigning "/boot" to the internal flash memory within the Debian installer would just put minimal installation files there, instruct the boot loader (GRUB) to use these and put the remaining stuff elsewhere?
(just asking hypothetically. I have not yet decided, if I want to go that way. First I need to find out, how to recover, if I completely mess things up)

[Aki]

During my experiments with my NAS I also installed Debian on the local Hard Disk and booted from there. This should be completely fine for the BIOS or HW, but I had the same issue, that 4 out of 10 boots were unsuccessful.

Just for the record, can you please share a system log of a successful boot with the quiet parameter replaced by the debug parameter ?

You can use the following command from a console to gather the log:

Code: Select all

script log.txt
su -l -c "journalctl -b --no-pager"
exit

The log will be stored in the file named log.txt . Then, you can report the log in one of the following ways:

• paste it into the Debian Pastezone service (https://paste.debian.net) and report the URL/link of the paste in a follow-up message

• attaching the log to a follow-up forum message as a compressed zip or gz file

• pasting the log into the body of a follow-up message between code tags (if they fit the size of a forum message)

[quack]

here you go:

Since it wasn't clear to me, if you wanted specifically the log from booting from HDD or the "normal" boot from USB stick, I attached both.

USB-Boot.zip

(121.3 KiB) Downloaded 15 times

HDD-Boot.zip

(131.99 KiB) Downloaded 13 times

[quack]

Some unexpected news:

Out of curiosity I installed the previous version of Debian (V11).
Guess what: 10 out of 10 boots were successful.

Although I still want to make some more tests,
it seems this problem is only with the actual V12 of Debian.

Edit:
I did some more tests and have NOT seen that behavior with Debian V11.
As far as I can judge, this phenomenon is local to Debian V12.