No-boot kernel, working lvm in initramfs, volumes not found

Kernels & Hardware, configuring network, installing services

No-boot kernel, working lvm in initramfs, volumes not found

Postby timbgo » 2013-07-06 22:42

Searched, but didn't find issues close enough to mine.
I guess most of those who could advise on the problem further below, will know fine the whereabouts of the issue from these history lines (I'll add a comment or two starting with #):
Code: Select all
 504  apt-get source linux
  506  cd linux-3.9.6/
# just 'debug-info: true' into 'debug-info: false':
  507  vi debian/config/amd64/defines
# the old working kernel config to use:
  509  cp -aiv /boot/config-3.2.0-4-amd64  .config
  510  make menuconfig
  512  fakeroot make deb-pkg

That got me the following:
Code: Select all
root@debinv35:/Cmn/src_1# ls -l linux*.deb
-rw-r--r-- 1 root root   8382258 Jul  6 21:22 linux-headers-3.9.6-130706_3.9.6-130706-1_amd64.deb
-rw-r--r-- 1 root root 487521996 Jul  6 21:44 linux-image-3.9.6-130706_3.9.6-130706-1_amd64.deb
-rw-r--r-- 1 root root    924730 Jul  6 21:22 linux-libc-dev_3.9.6-130706-1_amd64.deb
root@debinv35:/Cmn/src_1#


So I:
Code: Select all
dpkg -i *.deb

and that installed the new kernel into /boot.

But upon reboot, no booting.

Copying by hand.

Code: Select all
Loading, please wait...
 Volume group "vg_r" not found
 Skipping volume group vg_r
Unable to find LVM volume vg_r/root
 Volume group "vg_C" not found
Unable to find LVM volume vg_C/swap
Gave up waiting for root device.  Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
   - Check root= (did the system wait for the right device?)
 - Missing modules (cat /proc/module: ls /dev)
ALERT!  /dev/mapper/vg_r-root does not exist.  Dropping to a shell!
modprobe: module ehci-pci not found in modules.dep
modprobe: module ehci-hcd not found in modules.dep
modprobe: module uhci-hcd not found in modules.dep
modprobe: module ohci-hcd not found in modules.dep
modprobe: module usbhid not found in modules.dep


BusyBox v1.20.2 (Debian 1:1.20.0-8.1) built-in shell (ash)
Enter 'help' for a list of built-in commands.

/bin/sh: can't access tty; job control turned off
(initramfs) _

But, pls. have a look, I've been there before, and I bet those
vg_r-XXX devices are about to show up. I bet, I don't know yet.

Code: Select all
(initramfs) lvm
lvm> lvdisplay vg_r

... I'm not copying all, its' tedious, but that command showed

    /dev/vg_r/root
    /dev/vg_r/var
    /dev/vg_r/tmp
    /dev/vg_r/usr
just fine, complete listing, as they are...

/dev/vg_C is not intrinsic for booting the system. Not indulging.

Instead, I'll reboot into the same non-booting new kernel.
And back at the prompt, I continue investigating...

Code: Select all
(initramfs) cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.9.6130418-21 root=/dev/mapper/vg_r-root ro quiet
(initramfs)

And I'll try and get that rootdelay to some higher than the default
value (and what is the default, where do I find more to read on this
issue?), and, lemme see what else I have been advised to try...

Code: Select all
(initramfs) lvm
lvm> lvdisplay /dev/mapper/vg_r-root
 --- Logical volume ---
 LV Path               /dev/vg_r/root
...[snip]...

is just fine, and I don't think that needs changing in the command line...

Anyways, how is this done on Debian...
Is it,
    /etc/default/grub
and the manuals around it...

and then
Code: Select all
update-initramfs

and what else... ?

I'll try boot into the working (suppose still) kernel... Be back...

No, not yet, I'll try rootdelay from the command line in this no-booter
first.

So, what I did, and to no avail, is, I can copy what grub took from my input:

Code: Select all
(initramfs)
{You guessed, I still got a no-booter new kernel}
;-)

Code: Select all
(initramfs) cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.9.6130418-21 dolvm rootdelay=10 root=/dev/vg_r/root ro echo "Loading initial ramdisk ..."
(initramfs)


I know what I need to try. I'll replace the human-named root with its UUID, and be back.
Sure, I need to boot with the old kernel to find out which it is...

Or maybe not.

Anyways, I could probably install the stable 6 (six) or more months old kernel, but I'm a little despondent at the very fact that so old stuff seems to be the only option... so I haven't even tried that yet.

("What's the question?" pharisaic ones would ask.)
For those who wouldn't otherwise figure the question out, here it is:
In what freeking fashion way or manner do I get this kernle to work? Without more and more ounces of my nerves wrecked.
Please!
: :|
timbgo
 
Posts: 265
Joined: 2013-04-14 12:17

Re: No-boot kernel, working lvm in initramfs, volumes not fo

Postby timbgo » 2013-07-06 23:25

The procedures above, I have performed, the fakeroot, as normal user.
But I did find somewhere that it could be problems with permissions in such way, so I decided (mind: that was already an umptierth try above), to try it as root.
I haven't moved anywhere particurarly in my understanding, just, this time, I issued the
Code: Select all
fakeroot make deb-pkg

in the unpacked kernel source dir, as root.
And here I install them.:
Code: Select all
root@debinv35:/Cmn/src_1# dpkg -i *.deb
Selecting previously unselected package linux-headers-3.9.6-130706.
(Reading database ... 159064 files and directories currently installed.)
Unpacking linux-headers-3.9.6-130706 (from linux-headers-3.9.6-130706_3.9.6-130706-1_amd64.deb) ...
Selecting previously unselected package linux-image-3.9.6-130706.
Unpacking linux-image-3.9.6-130706 (from linux-image-3.9.6-130706_3.9.6-130706-1_amd64.deb) ...
dpkg: warning: downgrading linux-libc-dev from 3.9.6130418-21-1 to 3.9.6-130706-1
Preparing to replace linux-libc-dev 3.9.6130418-21-1 (using linux-libc-dev_3.9.6-130706-1_amd64.deb) ...
Unpacking replacement linux-libc-dev ...
Setting up linux-headers-3.9.6-130706 (3.9.6-130706-1) ...
Setting up linux-image-3.9.6-130706 (3.9.6-130706-1) ...
update-initramfs: Generating /boot/initrd.img-3.9.6-130706
Generating grub.cfg ...
Found background image: .background_cache.png
Found linux image: /boot/vmlinuz-3.9.6-130706
Found initrd image: /boot/initrd.img-3.9.6-130706
Found linux image: /boot/vmlinuz-3.2.43-grsec130418-21
Found initrd image: /boot/initrd.img-3.2.43-grsec130418-21
Found linux image: /boot/vmlinuz-3.2.0-4-amd64
Found initrd image: /boot/initrd.img-3.2.0-4-amd64
Found Windows Vista (loader) on /dev/sda4
done
Setting up linux-libc-dev (3.9.6-130706-1) ...
root@debinv35:/Cmn/src_1#

I'll now go and reboot. Be back.
timbgo
 
Posts: 265
Joined: 2013-04-14 12:17

Re: No-boot kernel, working lvm in initramfs, volumes not fo

Postby timbgo » 2013-07-07 05:24

Doing the same compilation as root made no difference.
Same (or similar) errors, still no boot.
timbgo
 
Posts: 265
Joined: 2013-04-14 12:17

Re: No-boot kernel, working lvm in initramfs, volumes not fo

Postby timbgo » 2013-07-07 08:49

I know it's dead time of the week, so no hurry.

I have posted here:
https://forums.grsecurity.net/viewtopic.php?f=3&t=3032&p=12873#p12873
a complaint and actually accusation

and here:
https://forums.grsecurity.net/viewtopic.php?f=3&t=3585&p=13189#p13189
and invitation to mend these vitriolisms, please!
timbgo
 
Posts: 265
Joined: 2013-04-14 12:17

Re: No-boot kernel, working lvm in initramfs, volumes not fo

Postby timbgo » 2013-07-08 09:10

Here's the .config in /linux-3.9.8 that got me the packages I listed above, upon
the install of which I got the:
/boot/initrd.img-3.9.8-grsec130705
which doesn't boot the system, here's the part of the config, pasting over from menuconfig help:
Code: Select all
  | CONFIG_BLK_DEV_DM:                                                             | 
  |                                                                                | 
  | Device-mapper is a low level volume manager.  It works by allowing             | 
  | people to specify mappings for ranges of logical sectors.  Various             | 
  | mapping types are available, in addition people may write their own            | 
  | modules containing custom mappings if they wish.                               | 
  |                                                                                | 
  | Higher level volume managers such as LVM2 use this driver.                     | 
  |                                                                                | 
  | To compile this as a module, choose M here: the module will be                 | 
  | called dm-mod.                                                                 | 
  |                                                                                | 
  | If unsure, say N.                                                              | 
  |                                                                                | 
  | Symbol: BLK_DEV_DM [=m]                                                        | 
  | Type  : tristate                                                               | 
  | Prompt: Device mapper support                                                  | 
  |   Defined at drivers/md/Kconfig:177                                            | 
  |   Depends on: MD [=y]                                                          | 
  |   Location:                                                                    | 
  |     -> Device Drivers                                                          | 
  |       -> Multiple devices driver support (RAID and LVM) (MD [=y]) 

If I grep all my current kernel configs:
Code: Select all
root@myhost:# cat /boot/config-3.* | grep 'BLK_DEV_DM='
config-3.2.0-4-amd64       config-3.2.43-grsec130418  config-3.9.8-grsec130705
root@myhost:# cat /boot/config-3.* | grep 'BLK_DEV_DM='
CONFIG_BLK_DEV_DM=m
CONFIG_BLK_DEV_DM=m
CONFIG_BLK_DEV_DM=y
root@myhost:#


And that is the only difference if I apply the wisdom explicated in:

http://wiki.debian.org/InitramfsDebug

apart from the version, if I diff, like suggested at the end.
There is no:
Code: Select all
./lib/modules/3.9.8-grsec130705/kernel/drivers/md/dm-mod.ko

while there is:
Code: Select all
./lib/modules/3.2.43-grsec130418/kernel/drivers/md/dm-mod.ko


But, how come? In the .config of 3.9.8 it is:
Code: Select all
Symbol: BLK_DEV_DM [=m]

it is not:
Code: Select all
Symbol: BLK_DEV_DM [=y]

so either I have messed up or some bug somewhere.
I have already tried to switch btwn =m and =y a couple of times, so I'll try
one more time, writing down exactly what I do and what I get in respect to this which
in the page that gives the above help, contains that item among these:
Code: Select all
  | |    --- Multiple devices driver support (RAID and LVM)                      | | 
...[snip]... 
| |    -M-     RAID-4/RAID-5/RAID-6 mode                                       | | 
  | |    <M>     Multipath I/O support                                           | | 
  | |    <M>     Faulty test module for MD                                       | | 
  | |    <M>   Device mapper support                                             | | 
  | |    [ ]     Device mapper debugging support                                 | | 
  | |    <M>     Crypt target support                                            | | 
...[snip]... 




I will try again, another recompile, but what disheartens me a little is
that the dm-mod.ko probably won't be produced for some reason, and then the initramfs won't find /dev/mapper/vg_r-root and the kernel won't boot, because:
( I just found I am wrong here, but let me finish my thought )
Code: Select all
root@myhost:# diff .config*
.config      .config.old 
root@myhost:# grep '_BLK_DEV_DM=' .config*
.config:CONFIG_BLK_DEV_DM=m
.config.old:CONFIG_BLK_DEV_DM=m
root@myhost:# ls -l .config*
-rw-r--r-- 1 root root 127071 Jul  8 07:35 .config
-rw-r--r-- 1 root root 127071 Jul  7 01:14 .config.old
root@myhost:# md5sum .config*
771cb20579702ae69926a4c379f2df9b  .config
771cb20579702ae69926a4c379f2df9b  .config.old
root@myhost:#


But:

Code: Select all
root@myhost:# md5sum .config /boot/config-3.9.8-grsec130705
771cb20579702ae69926a4c379f2df9b  .config
e7f0b09bfee62f65883a84b51e6d6484  /boot/config-3.9.8-grsec130705
root@myhost:# diff .config /boot/config-3.9.8-grsec130705
53c53
< CONFIG_LOCALVERSION="-130706"
---
> CONFIG_LOCALVERSION="130705"
1943c1943
< CONFIG_BLK_DEV_DM=m
---
> CONFIG_BLK_DEV_DM=y
root@myhost:# ls -l ../*.deb
-rw-r--r-- 1 root root   1136116 Jul  7 05:32 ../linux-firmware-image_3.9.8-grsec-130706-1_amd64.deb
-rw-r--r-- 1 root root   8345010 Jul  7 05:33 ../linux-headers-3.9.8-grsec-130706_3.9.8-grsec-130706-1_amd64.deb
-rw-r--r-- 1 root root 533657160 Jul  7 06:23 ../linux-image-3.9.8-grsec-130706_3.9.8-grsec-130706-1_amd64.deb
-rw-r--r-- 1 root root    925976 Jul  7 05:33 ../linux-libc-dev_3.9.8-grsec-130706-1_amd64.deb
root@myhost:#


I haven't yet tried to install these packages...
I got a little downcast at the state of GNU/Linux, and especially Debian, of today, and related issues, and haven't yet tried to install these latest packages.
I will now.
Be back...

Oh, just before I go, there is:
Code: Select all
./lib/modules/3.9.8-grsec-130706/kernel/drivers/md/dm-mod.ko


I booted:

Code: Select all
# uname -r
3.9.8-grsec130705
#


and lemme see if I remembered what changes I made to the kernel command line, to successfully boot.
In the first place, I obviously edited grub.cnf (in grub interface) of the older kernel. but it's grsecurity patched, so it's what I needed.

I thought to make it first with Debian testing kernel, to get support, but in 10 or 15 hours there was none... Issue same in both non-grsec and grsec kernels.

I'll save dmesg in separate file...

Useless. It only has saved from second 39 of the boot process (IIUC), showing top of the dmesg output:
[ 39.229693] grsec: exec of /usr/bin/tput (/usr/bin/tput civis ) by /usr/bin/tput[dbus:2633] uid/euid:0/0 gid/egid:0/0, parent /etc/init.d/dbus[dbus:2531] uid/euid:0/0 gid/egid:0/0
[ 39.232504] grsec: exec of /usr/bin/tput (/usr/bin/tput sc ) by /usr/bin/tput[dbus:2634] uid/euid:0/0 gid/egid:0/0, parent /etc/init.d/dbus[dbus:2531] uid/euid:0/0 gid/egid:0/0
[ 39.235974] grsec: exec of /usr/bin/tput (/usr/bin/tput hpa 0 ) by /usr/bin/tput[dbus:2635] uid/euid:0/0 gid/egid:0/0, parent /etc/init.d/dbus[dbus:2531] uid/euid:0/0 gid/egid:0/0
[ 39.238888] grsec: exec of /bin/echo (/bin/echo -ne [ [32m ok [39;49m ) by /bin/echo[dbus:2636] uid/euid:0/0 gid/egid:0/0, parent /etc/init.d/dbus[dbus:2531] uid/euid:0/0 gid/egid:0/0
[ 39.241192] grsec: exec of /usr/bin/tput (/usr/bin/tput rc ) by /usr/bin/tput[dbus:2637] uid/euid:0/0 gid/egid:0/0, parent /etc/init.d/dbus[dbus:2531] uid/euid:0/0 gid/egid:0/0
...[snip]...

First of all, since this tribulation with no-booting kernel might be over, here's what helped me, reading and comparing there:

https://bbs.archlinux.org/viewtopic.php?id=145714

Now I see that dolvm is anyway default at start of grub.cnf, so that didn't get it to boot...
No, not dolvm, but:
insmod lvm
but that means dolvm is superfluous.

On the other hand, I remember:
/dev/mapper/vg_r-root

really not being there in plain (initramfs) showing, which is busybox prompt, but only for lvm issued in there.

I added, that was in grub started busybox:
rootdelay=20 (or was it even 30)

Also I added:
lvmwait=/dev/mapper/vg_r-root

and I will have to figure the correct order that got me into booting.
Just don't remember now.

I think all that's missing, really, might be only one of either the lvmwait= line above, or rootdelay=<more than 10> line but in the format that fits in
/etc/default/grub
and which is the format?

What I will try now is make this change, this line:
Code: Select all
GRUB_CMDLINE_LINUX=""

change to:
Code: Select all
GRUB_CMDLINE_LINUX="lvmwait=/dev/mapper/vg_r-root rootdelay=20"


and reboot.

Can't believe I forgot that I wouldn't get nothing.
update-initramfs is needed...
first

This change is intrinsic, and it is documented in manuals and wikis and other places for Debian.
So this is just how it looks in practice, as I'm now quite confident this is the exact procedure for my occasion and the problems that I've had...


Code: Select all
root@myhost:/home/mr# cp /boot/grub/grub.cfg  /Cmn/BAK/grub.cfg_`date +%s`
[1]+  Done                    dhclient eth1
root@myhost:/home/mr# cp -v /boot/grub/grub.cfg  /Cmn/BAK/grub.cfg_`date +%s`
`/boot/grub/grub.cfg' -> `/Cmn/BAK/grub.cfg_1373283648'
root@myhost:/home/mr# cp -v /boot/grub/grub.cfg  /Cmn/BAK/grub.cfg_137328364
grub.cfg_1373283645  grub.cfg_1373283648 
root@myhost:/home/mr# rm /Cmn/BAK/grub.cfg_1373283645
root@myhost:/home/mr# rm /Cmn/BAK/grub.cfg_1373       
grub.cfg_1373109936  grub.cfg_1373111429  grub.cfg_1373283648 
root@myhost:/home/mr# rm /Cmn/BAK/grub.cfg_1373283648
root@myhost:/home/mr# cp -v /boot/grub/grub.cfg  /Cmn/BAK/grub.cfg_`date +%s`
`/boot/grub/grub.cfg' -> `/Cmn/BAK/grub.cfg_1373283672'
root@myhost:/home/mr# cp -v /boot/grub/grub.cfg  ^Cmn/BAK/                   
root@myhost:/home/mr# cp -v /boot/initrd.img-3.9.8-grsec-130706  /Cmn/BAK/_`date +%s`
initrd.img-3.9.8-grsec-130706
root@myhost:/home/mr# cp -v /boot/initrd.img-3.9.8-grsec-130706  /Cmn/BAK/initrd.img-3.9.8-grsec-130706_`date +%s`
`/boot/initrd.img-3.9.8-grsec-130706' -> `/Cmn/BAK/initrd.img-3.9.8-grsec-130706_1373283702'
root@myhost:/home/mr# update-initramfs -t -c -k 3.9.8-grsec-130706
update-initramfs: Generating /boot/initrd.img-3.9.8-grsec-130706
root@myhost:/home/mr# cp -v /boot/grub/grub.cfg  /Cmn/BAK/grub.cfg_`date +%s``/boot/grub/grub.cfg' -> `/Cmn/BAK/grub.cfg_1373283749'
root@myhost:/home/mr# cp -v /boot/initrd.img-3.9.8-grsec-130706  /Cmn/BAK/initrd.img-3.9.8-grsec-130706_`date +%s`
`/boot/initrd.img-3.9.8-grsec-130706' -> `/Cmn/BAK/initrd.img-3.9.8-grsec-130706_1373283756'
root@myhost:/home/mr# ls -l /Cmn/BAK/*_1373283*
grub.cfg_1373283672                       initrd.img-3.9.8-grsec-130706_1373283702
grub.cfg_1373283749                       initrd.img-3.9.8-grsec-130706_1373283756
root@myhost:/home/mr# ls -l /Cmn/BAK/*_1373283*
-r--r--r-- 1 root root     6266 Jul  8 11:41 /Cmn/BAK/grub.cfg_1373283672
-r--r--r-- 1 root root     6266 Jul  8 11:42 /Cmn/BAK/grub.cfg_1373283749
-rw-r--r-- 1 root root 11631459 Jul  8 11:41 /Cmn/BAK/initrd.img-3.9.8-grsec-130706_1373283702
-rw-r--r-- 1 root root 11631472 Jul  8 11:42 /Cmn/BAK/initrd.img-3.9.8-grsec-130706_1373283756
root@myhost:/home/mr# ls -l /Cmn/BAK/g*_1373283*
grub.cfg_1373283672  grub.cfg_1373283749 
root@myhost:/home/mr# diff /Cmn/BAK/g*_1373283*
root@myhost:/home/mr# grub-mkconfig -o /boot/grub/grub.cfg
Generating grub.cfg ...
Found background image: /usr/share/images/desktop-base/desktop-grub.png
Found linux image: /boot/vmlinuz-3.9.8-grsec-130706
Found initrd image: /boot/initrd.img-3.9.8-grsec-130706
Found linux image: /boot/vmlinuz-3.9.8-grsec130705
Found initrd image: /boot/initrd.img-3.9.8-grsec130705
Found linux image: /boot/vmlinuz-3.2.43-grsec130418
Found initrd image: /boot/initrd.img-3.2.43-grsec130418
Found linux image: /boot/vmlinuz-3.2.0-4-amd64
Found initrd image: /boot/initrd.img-3.2.0-4-amd64
Found Windows XP Professional x64 Edition on /dev/sda4
done
root@myhost:/home/mr#



That was real time.


And I was right to be confident.
Even though I still anxiously prayed to God and His Mother during the loong 20 seconds delay.
This is solved!
Oh, and if someone with insufficient understanding and skills, lack of understanding of how stupid is to entrust your computer to NSA's SELinux, which I figure is lots of people in the US and satellites regimes worldwide, lack of skills to compile GNU/Linux kernel patched with Grsecurity/Pax first (oh no, wrong, it's a no brainer: https://en.wikibooks.org/wiki/Grsecurity, or rather it's a same-brainer as any kernel compile, there's Debian-wise: https://en.wikibooks.org/wiki/Grsecurity/Configuring_and_Installing_grsecurity#On_Debian_and_Ubuntu there too, i.e. first patched with those most important programs for anybody's privicy whosoever...
And if someone is insuffient in anctually only understanding of how stupid it is to hand your computer over to NSA through SELinux and associates (if this post survives, even though some vitrioloids have infiltrated and are seemingly even taking over Debian, the Christian Marillat's persecution (he can't be not findable by the scum search engine everybody uses, can he?) being obvious testimony to that...
And if someone is insuffient and actually persistent in using non-Grsecurity patched kernel, and this post of mine survives, I am willing to test whether this procedure works for SELinux regular testing Debian 3.9.6 kernel.

Thank you!
timbgo
 
Posts: 265
Joined: 2013-04-14 12:17

Re: No-boot kernel, working lvm in initramfs, volumes not fo

Postby shadowman32 » 2016-04-06 14:09

troll yourself, you dont ya!

Its an improvement, just like USN made TOR. So by your definition, TOR is flawed because US GOVT made it. The NSA has half a brain cell. Mebbe they dont know crypto as well as others, but they have half a brain cell, which for a govt agency says a lot.

Nothing wrong with grsecurity, assuming you can patch it to work.
And you didnt explain anything.

You tripped over your grub half a dozen times.
Looks like you forgot to depmod and update your initrd. If its missing from your depends file, check it, mod it, then rebuild the initrd.
We arent impressed by your tin foil hat.

But im in same boat, LVM fails because the MODULE isnt TOLD to load. Custom 4.4.6. And NO, GRSEC wont patch correctly.
shadowman32
 
Posts: 8
Joined: 2015-04-16 18:52


Return to System configuration

Who is online

Users browsing this forum: No registered users and 9 guests

fashionable