[SOLVED] LVM: lv not available on boot after system update

Kernels & Hardware, configuring network, installing services

[SOLVED] LVM: lv not available on boot after system update

Postby mauromol » 2018-04-05 19:19

I have an old wheezy installation (on a VM) I use to cross-compile software for an ARM box.
Today I did an "aptitude update" followed by "aptitude upgrade" to refresh the installed packages (there were mainly security updates).
However, after I installed all the updates and rebooted, now the boot process (whichever kernel I choose, 3.2.0-4 or 3.2.0-5) will stop with the following reason:

Gave up waiting for root device. Common problems:
[cut]
ALERT! /dev/disk/by-uuid/fa6... does not exist
Dropping to a shell!


and the initramfs prompt from BusyBox is shown.

The root mount point is on a LVM logical volume.
On the initramfs I type "lvm", followed by "lvdisplay" and this is what is shown:

Code: Select all
LV Path   /dev/mycloud-crosscompile/root
LV Name root
VG Name mycloud-crosscompile
[...]
LV Status NOT available
[...]

LV Path   /dev/mycloud-crosscompile/swap_1
LV Name swap_1
VG Name mycloud-crosscompile
[...]
LV Status available


As you can see, two logical volumes created on the same Volume Group, but one of them (the root one) is NOT available.

/etc/fstab correctly tries to mount the root from the right device: /dev/mycloud-crosscompile/root
(and not by UUID, as the boot error message seems to suggest)

vgdisplay shows:

Code: Select all
VG Name mycloud-crosscompile
System ID (empty)
Format lvm2
[...]
MAX LV 0
Cur LV 2
Open LV 0
Max PV 0
Cur PV 2
Act PV 2


pvdisplay:

Code: Select all
PV Name /dev/sda5
VG Name mycloud-crosscompile
[...]

PV Name /dev/sdb1
VG Name mycloud-crosscompile
[...]


So, all seems to be fine, except from the root logical volume being NOT available.

Running "vgchange -ay" shows:

Code: Select all
2 logical volume(s) in volume group "mycloud-crosscompile" now active.


Then I type "exit" twice (once to exit "lvm" prompt, once to exit the "initramfs" prompt) and then boot starts and completes normally.

I spent a lot of time today to search for others having similar problems, but nobody seems to be in my same situation. What I understand is that probably LVM initialisation is not completed soon enough for the boot sequence to find the root logical volume. But I don't understand why and especially why it started to happen only after a system update.

I already tried to type again "vgchange -ay" or "update-initramfs -uv", but with no luck: next reboot will fail with the exact same problem.

Any help would be greatly appreciated.

Mauro.
Last edited by mauromol on 2018-04-06 14:10, edited 1 time in total.
mauromol
 
Posts: 20
Joined: 2013-01-19 16:38

Re: LVM: lv not available on boot after system update

Postby p.H » 2018-04-05 20:53

"LV not available" just means that the LV has not been activated yet, because the initramfs does not feel like it should activate the LV.

The problem seems to be in the root= parameter passed by GRUB to the kernel command line as defined in /boot/grub/grub.cfg. When the root file system is on an LV, the root parameter should be /dev/mapper/VGname-LVname instead of the UUID, so that the initramfs knows it is an LV and can activate and mount it. Try to run update-grub again and see if that fixes the problem.
p.H
 
Posts: 344
Joined: 2017-09-17 07:12

Re: LVM: lv not available on boot after system update

Postby mauromol » 2018-04-06 09:54

Hi!
Thank you for your help.
If I run "update-grub" as root, I get the following error:

Code: Select all
/usr/sbin/grub-probe: error: Couldn't find PV pv1. Check your device.map.


I have always changed GRUB configuration just in /etc/default/grub, but I can't find any reference to PV pv1 there. I don't know what is device.map.

Thanks again!
mauromol
 
Posts: 20
Joined: 2013-01-19 16:38

Re: LVM: lv not available on boot after system update

Postby p.H » 2018-04-06 11:07

AFAIK LVM PVs do not have a name or label, unlike VGs and LVs.

device.map is a file usually located in /boot/grub which contains the translation map between Linux drive names (such as /dev/sda) and GRUB drive names (such as hd0). But I thought that it was somehow deprecated and optional.

It seems that the VG mycloud-crosscompile spans on two PVs in partitions /dev/sda5 and /dev/sdb1 located on separate disks.

Maybe update-grub expects to find the two disks in /boot/grub/device.map. You can check its contents and update it with grub-mkdevicemap.
p.H
 
Posts: 344
Joined: 2017-09-17 07:12

Re: LVM: lv not available on boot after system update

Postby mauromol » 2018-04-06 14:10

Thanks to your help, I was put on the right track.
I found this bug report: https://bugs.debian.org/cgi-bin/bugrepo ... bug=697592
which was archived because tracked only for squeeze, but evidently still applicable to wheezy.

The solution was to move away /boot/grub/device.map and re-run "update-grub".
This created a new /boot/grub/device.map with two drives listed instead of one.
I don't know why this was not necessary before the upgrade.
Anyway yes, I have two "physical" disks (virtual ones, indeed) added to the same VG and the root LV probably spans on them both.

It's interesting that the contents of /boot/grub/device.map is:
Code: Select all
(hd0) /dev/disk/by-id/ata-VBOX_HARDDISK_VB179...
(hd1) /dev/disk/by-id/ata-VBOX_HARDDISK_VB12f...


and it is not mentioning LVM paths... This is probably the main source of the problem.
mauromol
 
Posts: 20
Joined: 2013-01-19 16:38

Re: [SOLVED] LVM: lv not available on boot after system upda

Postby p.H » 2018-04-06 14:44

mauromol wrote:The solution was to move away /boot/grub/device.map and re-run "update-grub".

Did it also fix the root= value in the kernel command line and solve the boot problem ?

mauromol wrote:I don't know why this was not necessary before the upgrade.

Maybe because no system update had required to run update-grub so far. AFAIK only kernel updates do.

mauromol wrote:It's interesting that the contents of /boot/grub/device.map (...) is not mentioning LVM paths.

This is normal. device.map contains the map between raw BIOS (or UEFI) drives and OS devices. The BIOS or UEFI firmware knows nothing about logical devices such as partitions, LVM volumes or RAID arrays.
p.H
 
Posts: 344
Joined: 2017-09-17 07:12

Re: [SOLVED] LVM: lv not available on boot after system upda

Postby mauromol » 2018-04-06 15:33

p.H wrote:
mauromol wrote:The solution was to move away /boot/grub/device.map and re-run "update-grub".

Did it also fix the root= value in the kernel command line and solve the boot problem ?


Yes, in /boot/grub/brug.cfg now the root= kernel boot parameter points to /dev/mapper/mycloud-crosscompile-root and now the system boots with no problem!

p.H wrote:
mauromol wrote:I don't know why this was not necessary before the upgrade.

Maybe because no system update had required to run update-grub so far. AFAIK only kernel updates do.


So, you think that the corruption was made by running "update-grub" after the kernel update, because before this happened the root= kernel boot parameter was already set up correctly, even if /boot/grub/device.map was still wrong?
Perhaps it went in this way, however I think this is a serious problem in wheezy.
It's interesting that in my main Mint system (based on Ubuntu 14.04), where I still use LVM, I don't have /boot/grub/device.map file at all. But this system boots with UEFI, so it probably works in another way.

Thanks again for your assistance!
mauromol
 
Posts: 20
Joined: 2013-01-19 16:38

Re: [SOLVED] LVM: lv not available on boot after system upda

Postby p.H » 2018-04-07 07:53

mauromol wrote:So, you think that the corruption was made by running "update-grub" after the kernel update, because before this happened the root= kernel boot parameter was already set up correctly, even if /boot/grub/device.map was still wrong?

I can only suspect. I tried to reproduce the issue. If a drive is missing in device.map, update-grub spits the error about a missing PV, but the generated grub.cfg is different : the root= kernel parameter still points to /dev/mapper/... but all commands needed to locate and access the vmlinuz and initrd files are missing, so I guess they would not even be loaded.

mauromol wrote:It's interesting that in my main Mint system (based on Ubuntu 14.04), where I still use LVM, I don't have /boot/grub/device.map file at all. But this system boots with UEFI, so it probably works in another way.

UEFI boot is not different after the GRUB core image has been loaded. But I observed that Ubuntu puts more files than Debian in /boot/efi.

Also, the GRUB version is Wheezy is older (1.9x) than in later releases (2.x) and works a bit differently.
p.H
 
Posts: 344
Joined: 2017-09-17 07:12


Return to System configuration

Who is online

Users browsing this forum: No registered users and 2 guests

fashionable