Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

problems with Xen and Bullseye upgrade

Linux Kernel, Network, and Services configuration.
Post Reply
Message
Author
Camdav
Posts: 7
Joined: 2021-11-16 02:11

problems with Xen and Bullseye upgrade

#1 Post by Camdav »

I had a Xen/Debian 10 system with 3 DomU VMs all running Debian 10 more or less smoothly.
  • No graphics hardware, beyond the basic console; no gui
  • All DomU's tested are based on a single LVM volume as the root filesystem and another as swap.
  • All use bridged LANs, some connected to a real nic, some not.
  • the DomU system I have not upgraded has a pci passthrough for the USB controller.
  • hardware i5-9600K, Z390 chipset, 16GB ram
I upgraded the least critical VM (email gateway) to Debian 11 as my first test. Everything seemed to function OK, except, when I went to reboot, it never completed shutown and it required a manual "xl destroy .." to get rid of it.
I cannot get a system to shutdown either by a shutdown/reboot command within the VM, nor by "xl shutdown/reboot" issued from Dom0.

In a spirit of adventure I upgraded a second VM - with the same result.

I ran up a new bullseye installation on a spare old PC and having both Dom0 and Domu as fresh Bullseye installs did not show the shutdown problems.

So, I figured, upgrade Dom0 to Bullseye, but the problem was not fixed.
I created a fresh minimal DomU on the old system and the shutdown still fails, but with slightly different information shown:

On the working fresh system (reboot from a console login):
journal records "[ OK ] Reached target Power-Off"
systemd-shutdown[1] Syncing filesystems
systemd-shutdown[1] sending SIGTERM to remaining processes ...
then console shows journald received SIGTERM (and nothing more is in journal)
root filesystem is remounted r/o
systemd-shutdown[1] shows about 16 entries listing the shudown process, finishing with..
"systemd-shutdown[1]: Powering off" and
"reboot:system halted"
On the failing system that was a fresh DomU install, I can run xl shutdown and then any time later I can open the console with
xl console <dom>.
The console displays the same as above, getting to
"systemd-shutdown[1]: Powering off",
but never the final "system halted".
So I can go in and out of console view until I destroy the VM.


On the failing systems that were upgraded the console shows no messages from systemd-shutdown[1], although it looks to be getting to the same place because is does show the message about journald shutting down and one line about the root filesystem being remounted.

Conclusion so far - the common factor is my Dom0 (either buster or bullseye) together with bullseye on the DomU; either:
  1. some old configuration setting has come back to bite me, or
  2. something about the hardware is not compatible?
Any suggestions where to look next?

raumfahrer
Posts: 1
Joined: 2022-02-12 19:52

Re: problems with Xen and Bullseye upgrade

#2 Post by raumfahrer »

Hi Camdav,

I can confirm that I am having very similar issues. I also updated from Buster to Bullseye and I am experiencing DomU shutdown failures. However, it seems that at least some newly created DomUs can shutdown completely. I can't find the reason for the different behaviour, though. A newly created Bullseye PV DomU (created with the debootstrap procedure described in https://wiki.xenproject.org/wiki/Xen_Pr ... ners_Guide) can't shutdown and stays at

Code: Select all

systemd-shutdown[1]: Powering off.
There is a similar issue described in https://www.mail-archive.com/xen-devel@ ... 92490.html.

szlovakm
Posts: 1
Joined: 2022-02-19 20:37

Re: problems with Xen and Bullseye upgrade

#3 Post by szlovakm »

Hi,

I can confirm too, I am having the same issues.
But...
I tested on an old PC (more then 12 years old, no UEFI, only BIOS), New installation, Dom0: Bullseye, DomU: Bullseye -- there was no problem.
I tested on a newer PC (about 5 years old), New installation, Dom0: Bullseye, DomU: Bullseye -- there was the same problem. Either BIOS boot, or UEFI boot. (First I installed in legacy mode, later I installed in UEFI mode.)
IMHO the DomU stucks on poweroff or reboot depends on hardware, not on upgrade / clean install.

CwF
Global Moderator
Global Moderator
Posts: 2625
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 190 times

Re: problems with Xen and Bullseye upgrade

#4 Post by CwF »

szlovakm wrote: 2022-02-19 20:49 IMHO the DomU stucks on poweroff or reboot depends on hardware, not on upgrade / clean install.
Yes, that's why even Dom0 itself starts life as a nested vm. It is imbued with configuration to adapt to the known conditions and hardware it will run on. Then, an issue is narrowed is scope since we know the 'image' runs well here, there, and virtualized. I resist unique installs as much as possible unless I want a full reset to start over, hasn't happened.

Camdav
Posts: 7
Joined: 2021-11-16 02:11

Re: problems with Xen and Bullseye upgrade

#5 Post by Camdav »

szlovakm wrote: 2022-02-19 20:49 IMHO the DomU stucks on poweroff or reboot depends on hardware, not on upgrade / clean install.
My experience so far is that it is a combination of both hardware and software version.
The DomU that gave problems had retained a version 4 kernel from Buster and if I booted the DomU to that old kernel then it did not show this problem.
The issue you reported from xen-devel is very similar, although my experience with kernel versions might be different.

I am seeing some suspicious kernel log messages during boot in the problematic systems, but I have not found time to investigate further.

Camdav
Posts: 7
Joined: 2021-11-16 02:11

Re: problems with Xen and Bullseye upgrade - fixed?

#6 Post by Camdav »

With the recent upgrades the problem has been resolved.

kernel: linux-image-5.10.0-20-amd64 (5.10.158-2 (2022-12-13))
xen packages: 4.14.5+86

These were all updated around the same time for Dom0 and DomU.

Post Reply