[Solved] update-grub (lsblk) not detecting LVs as expected

[ksu]

running update-grub suddenly started failing to detect other LVs: ( all LVs used to get activated automatically on boot )

Code: Select all

deb10 ~ # update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.1.0-20-amd64
Found initrd image: /boot/initrd.img-6.1.0-20-amd64
Found linux image: /boot/vmlinuz-6.1.0-18-amd64
Found initrd image: /boot/initrd.img-6.1.0-18-amd64
Found linux image: /boot/vmlinuz-6.1.0-17-amd64
Found initrd image: /boot/initrd.img-6.1.0-17-amd64
Found linux image: /boot/vmlinuz-6.1.0-10-amd64
Found initrd image: /boot/initrd.img-6.1.0-10-amd64
Found linux image: /boot/vmlinuz-6.1.0-9-amd64
Found initrd image: /boot/initrd.img-6.1.0-9-amd64
Adding boot menu entry for UEFI Firmware Settings ...
Found memdisk: /memdisk
Imagepath /boot/images not found
Warning: os-prober will be executed to detect other bootable partitions.
Its output will be used to detect bootable binaries on them and create new boot entries.
lsblk: /dev/mapper/spvg1-root: not a block device
lsblk: /dev/mapper/spvg1-var: not a block device
lsblk: /dev/mapper/vg1-root: not a block device
lsblk: /dev/mapper/vg1-varlog: not a block device
Found Windows Boot Manager on /dev/nvme1n1p1@/EFI/Microsoft/Boot/bootmgfw.efi
done
deb10 ~ # uname -a
Linux deb10 6.1.0-20-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.85-1 (2024-04-11) x86_64 GNU/Linux
deb10 ~ # 

LVM:

Code: Select all

deb10 ~ # vgs
  VG    #PV #LV #SN Attr   VSize    VFree  
  linvg   1   3   0 wz--n-  171.82g <16.20g
  spvg1   1   2   0 wz--n- <100.47g <51.47g
  vg1     1   2   0 wz--n-  666.63g 259.91g
deb10 ~ # lvs
  LV      VG    Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  linlog  linvg -wi-ao----   4.00g                                                    
  linroot linvg -wi-ao---- 139.62g                                                    
  rootsrc linvg -wi-a-----  12.00g                                                    
  root    spvg1 -wi-------  45.00g                                                    
  var     spvg1 -wi-------   4.00g                                                    
  root    vg1   -wi------- 403.00g                                                    
  varlog  vg1   -wi-------   3.72g                          
  

LSBLK:

Code: Select all

deb10 ~ # lsblk
NAME              MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
nvme1n1           259:0    0   1.8T  0 disk 
├─nvme1n1p1       259:1    0   500M  0 part /boot/efi
├─nvme1n1p2       259:2    0 892.8G  0 part 
├─nvme1n1p3       259:3    0   863M  0 part 
├─nvme1n1p4       259:4    0   787G  0 part 
├─nvme1n1p5       259:5    0    10G  0 part /boot
└─nvme1n1p6       259:6    0 171.8G  0 part 
  ├─linvg-linroot 254:2    0 139.6G  0 lvm  /
  ├─linvg-linlog  254:5    0     4G  0 lvm  /var/log
  └─linvg-rootsrc 254:6    0    12G  0 lvm  
nvme0n1           259:7    0   1.8T  0 disk 
├─nvme0n1p1       259:8    0    10G  0 part 
├─nvme0n1p2       259:9    0 666.6G  0 part 
├─nvme0n1p3       259:10   0   512M  0 part 
├─nvme0n1p4       259:11   0 100.5G  0 part 
├─nvme0n1p5       259:12   0   299M  0 part 
└─nvme0n1p6       259:13   0    16M  0 part 

a workaround is a manual activation before running update-grub ( this was not required before )

Code: Select all

deb10 ~ # vgchange -ay
  2 logical volume(s) in volume group "spvg1" now active
  2 logical volume(s) in volume group "vg1" now active
  3 logical volume(s) in volume group "linvg" now active

troubleshooting done:
i have downgraded the recently updated util-linux package back to:

Code: Select all

util-linux-extra_2.38.1-5+b1_amd64.deb
util-linux-locales_2.38.1-5_all.deb
util-linux_2.38.1-5+b1_amd64.deb

recreated all the initramfs ( should recreate just one )
no improvement after rebooting only the main/booted linvg LVs are active, update-grub fails
there is no recent changes applied that I'm aware of that could influence the way update-grub "detects" the filesystems, or LVM scan/activation during boot.
Other than util-linux, gcc, and and 6.1.0-20-amd64 kernel update applied recently...
booting the older kernels results in the same problem ( but, i've recreated all initramfs's - by mistake - making the comparison between kernels and the boot process invalid )

i do not remember seeing update-grub/lsblk failing like this before before

thanks all!

[ksu]

more troubleshooting:
- analyzed lvm syslog from past ~4 weeks - lvm2-monitor and lvm2-lvmpolld starting as expected - no errors related to block storage/filesystems, etc...
- nothing custom in /etc/lvm2
- nothing unusual found in /etc/default/grub, no customizations to /etc/initramfs-tools on this system
- booting off a deb12 custom livecd activates all VGs/LVs as expected: automatically during boot
- no unwanted ESC/special chars in the configs

i have done some testing with multiple custom builds of 6.7 kernel for bcachefs ( but not within last 1 month )
then a few days ago i saw this ( Mr Torvalds about 6.9 kernel ):

"Ok, so this rc3 looks a bit different than the usual ones, because there's a large series to bcachefs to do filesystem repair after corruption. Not normally something we'd see in an rc kernel, but hey, if you had a corrupted bcachefs filesystem you'd probably want this, and if you thought bcachefs was stable already, I have a bridge to sell you. Special deal only for you, real cheap.

i am thinking now that lvm2 got somehow corrupted preventing it from proper activation during boot
but then i would expect it to fail as well when 'vgchange -ay' is executed - plus i do not have any bcache fs that is expected to be mounted - there is one but inactive, for testing only

still looking...

[ksu]

what a relief
to anyone who experiments with xo-server: (smh)

Code: Select all

xo-server[1590]: 2024-04-14T12:10:53.644Z xo:mixins:hooks WARN start failure {
xo-server[1590]:   error: Error: Command failed with exit code 5: vgchange -an
xo-server[1590]:     Logical volume linvg/linroot contains a filesystem in use.
xo-server[1590]:     Can't deactivate volume group "linvg" with 2 open logical volume(s)
xo-server[1590]:     0 logical volume(s) in volume group "spvg1" now active
xo-server[1590]:     0 logical volume(s) in volume group "vg1" now active
xo-server[1590]:       at makeError (file:///opt/xo/xo-builds/xen-orchestra-202404051924/packages/xo-server/node_modules/execa/lib/error.js:60:11)
xo-server[1590]:       at handlePromise (file:///opt/xo/xo-builds/xen-orchestra-202404051924/packages/xo-server/node_modules/execa/index.js:124:26) {
xo-server[1590]:     shortMessage: 'Command failed with exit code 5: vgchange -an',
xo-server[1590]:     command: 'vgchange -an',
xo-server[1590]:     escapedCommand: 'vgchange -an',
xo-server[1590]:     exitCode: 5,
xo-server[1590]:     signal: undefined,
xo-server[1590]:     signalDescription: undefined,
xo-server[1590]:     stdout: '  0 logical volume(s) in volume group "spvg1" now active\n' +
xo-server[1590]:       '  0 logical volume(s) in volume group "vg1" now active',
xo-server[1590]:     stderr: '  Logical volume linvg/linroot contains a filesystem in use.\n' +
xo-server[1590]:       `  Can't deactivate volume group "linvg" with 2 open logical volume(s)`,
xo-server[1590]:     cwd: '/opt/xo/xo-builds/xen-orchestra-202404051924/packages/xo-server',
xo-server[1590]:     failed: true,
xo-server[1590]:     timedOut: false,
xo-server[1590]:     isCanceled: false,
xo-server[1590]:     killed: false
xo-server[1590]:   }
xo-server[1590]: }