[SOLVED] Cannot access grub.cfg-UEFI boot Buster on ZFS root

[tibdm]

Hello,

I've successfully managed to install the latest currently available Buster distro (frozen - 2019-03-11) on a ZFS root, booting in UEFI. This, in itself, already makes me rather happy

Unfortunately, I'm still facing a problem that I so far was unable to resolve by myself, despite searching for hours, not to say days. This is the reason I'm finally posting here in hope that someone more enlightened than I am might help me find a solution...

Here is the situation:

On a UEFI system, Debian Buster (frozen-20190311) has been installed on a single NVMe drive, using two partitions, one for the ESP, the other one being dedicated for ZFS.
The ZFS configuration / installation was mainly carried out following the GitHub's zfsonlinux HOWTO for Debian Stretch https://github.com/zfsonlinux/zfs/wiki/ ... oot-on-ZFS.

Therefore, the configuration of the disk, its partitions, ZFS datasets and /etc/fstab file are as follow:

Code: Select all

$ blkid
/dev/nvme0n1: LABEL="rpool" UUID="980046881818473778" UUID_SUB="11196991732143542666" TYPE="zfs_member" PTUUID="40008bd4-3d2a-4f78-8c32-2f48ad9873d4" PTTYPE="gpt"
/dev/nvme0n1p1: LABEL="rpool" UUID="980046881818473778" UUID_SUB="11196991732143542666" TYPE="zfs_member" PARTUUID="845abebe-7785-4785-8382-3761982fbcd6"
/dev/nvme0n1p3: LABEL_FATBOOT="EFI" LABEL="EFI" UUID="8241-499B" TYPE="vfat" PARTUUID="581b4260-b5c1-4389-8e11-aa43f71d35b9"


$ lsblk -f
NAME        FSTYPE     LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINT
nvme0n1     zfs_member rpool 980046881818473778                                  
├─nvme0n1p1 zfs_member rpool 980046881818473778                                  
└─nvme0n1p3 vfat       EFI   8241-499B                             507.3M     1% /boot/efi


$ zfs list -r -o name,mountpoint,canmount,exec,setuid rpool
NAME                            MOUNTPOINT  CANMOUNT  EXEC  SETUID
rpool                           /                off    on      on
rpool/ROOT                      none             off    on      on
rpool/ROOT/debian-10            /             noauto    on      on
rpool/ROOT/debian-10/opt        /opt              on    on      on
rpool/ROOT/debian-10/var        /var             off   off     off
rpool/ROOT/debian-10/var/cache  /var/cache        on   off     off
rpool/ROOT/debian-10/var/log    legacy            on   off     off
rpool/ROOT/debian-10/var/tmp    legacy            on    on     off
rpool/home                      /home             on    on     off
rpool/home/root                 /root             on    on     off
rpool/tmp                       legacy            on    on     off

$ cat /etc/fstab
# EFI BOOT PARTITION
PARTUUID=581b4260-b5c1-4389-8e11-aa43f71d35b9   /boot/efi   vfat   noatime,nofail,x-systemd.device-timeout=1   0   1

# ZFS FILESYSTEM LEGACY MOUNTS
rpool/ROOT/debian-10/var/log   /var/log   zfs   noatime,nodev,noexec,nosuid   0   0
rpool/ROOT/debian-10/var/tmp   /var/tmp   zfs   noatime,nodev,nosuid          0   0
rpool/tmp                      /tmp       zfs   noatime,nodev,nosuid          0   0

Once the system installation was completed, time had come for the GRUB(2) installation, which was executed as follow:

Code: Select all

$ apt install --yes grub-efi-am64
$ grub-probe /
zfs
$ update-initramfs -u -k all
$ update-grub
$ grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=debian-10 --recheck --no-floppy
... No error reported
$ ls /boot/grub/*/zfs.mod
/boot/grub/x86_64-efi/zfs.mod

At this point, the content of /boot becomes:

Code: Select all

$ ls -l /boot/
total 32823
-rw-r--r-- 1 root root   206411 Jan 17 19:56 config-4.19.0-2-amd64
drwxr-xr-x 3 root root     4096 Jan  1  1970 efi
drwxr-xr-x 5 root root        9 Mar 18 09:59 grub
-rw-r--r-- 1 root root 27223819 Mar 18 08:49 initrd.img-4.19.0-2-amd64
-rw-r--r-- 1 root root  3345492 Jan 17 19:56 System.map-4.19.0-2-amd64
-rw-r--r-- 1 root root  5180704 Jan 17 19:56 vmlinuz-4.19.0-2-amd64

ls -l /boot/efi/EFI/debian-10
total 3772
-rwxr-xr-x 1 root root   72144 Mar 18 09:27 fbx64.efi
-rwxr-xr-x 1 root root     113 Mar 18 09:27 grub.cfg
-rwxr-xr-x 1 root root 1435040 Mar 18 09:27 grubx64.efi
-rwxr-xr-x 1 root root 1168464 Mar 18 09:27 mmx64.efi
-rwxr-xr-x 1 root root 1169528 Mar 18 09:27 shimx64.efi

$ ls -l /boot/grub/
total 1584
drwxr-xr-x 2 root root       3 Mar 15 12:40 fonts
-r--r--r-- 1 root root    5675 Mar 18 10:14 grub.cfg
-rw-r--r-- 1 root root    1024 Mar 15 12:40 grubenv
drwxr-xr-x 2 root root      40 Mar 18 09:27 locale
-rw-r--r-- 1 root root 2396122 Mar 15 12:40 unicode.pf2
drwxr-xr-x 2 root root     270 Mar 18 09:27 x86_64-efi

The interesting part of /boot/grub/grub.cfg being:

Code: Select all

$ cat /boot/grub/grub.cfg
...
menuentry 'Debian Buster GNU/Linux' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-0d99d3724e04f532' {
	load_video
	insmod gzio
	if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
	insmod part_gpt
	insmod zfs
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root  0d99d3724e04f532
	else
	  search --no-floppy --fs-uuid --set=root 0d99d3724e04f532
	fi
	echo	'Loading Linux 4.19.0-2-amd64 ...'
	linux	/ROOT/debian-10@/boot/vmlinuz-4.19.0-2-amd64 root=ZFS=rpool/ROOT/debian-10 ro  
	echo	'Loading initial ramdisk ...'
	initrd	/ROOT/debian-10@/boot/initrd.img-4.19.0-2-amd64
}
...

While the content of /boot/efi/EFI/debian-10/grub.cfg is:

Code: Select all

search.fs_uuid 0d99d3724e04f532 root 
set prefix=($root)'/ROOT/debian-10@/boot/grub'
configfile $prefix/grub.cfg

Unfortunately, as stated at the beginning of this post, trying to boot the system with this in place immediately brings up the grub command line (grub>).
Which means grub was able to load but cannot locate the grub.cfg file...

Nevertheless, it is then quite possible to boot the system by typing one of the following sequences at the grub> prompt:

Code: Select all

grub> insmod part_gpt
grub> search --set --label rpool
grub> linux /ROOT/debian-10@/boot/vmlinuz-4.19.0-2-amd64 root=ZFS=rpool/ROOT/debian-10 ro
grub> initrd /ROOT/debian-10@/boot/initrd.img-4.19.0-2-amd64
grub> boot

Or:

Code: Select all

grub> search --label --set root rpool
grub> prefix=($root)'/ROOT/debian-10@/boot/grub'
grub> configfile $prefix/grub.cfg

This second one bringing us back to the Grub start menu that we'd so much like to reach immediately upon system startup !

I, obviously, tried to modify the /boot/efi/EFI/debian-10/grub.cfg file, inserting each of the above sequences, but to no avail
I read somewhere that this file was not used during the efi boot process, and that it was hardcoded into the sibling grubx64.efi executable. Which I was unable to confirm, even using GHex trying to explore it...

One thing I could determine, using efibootmgr, is that it is shimx64.efi that is called during the UEFI boot process:

Code: Select all

$ efibootmgr -v
BootCurrent: 0000
Timeout: 2 seconds
BootOrder: 0000
Boot0000* debian-10	HD(3,GPT,581b4260-b5c1-4389-8e11-aa43f71d35b9,0x800,0x100000)/File(\EFI\debian-10\shimx64.efi)

One other thing I wasn't able to figure out is: where does the UUID used in both grub.cfg come from ?
I've parsed all /dev/disk/by-id, zpool / zfs get guid and blkid output but couldn't determine where the 0d99d3724e04f532 value used in both /boot/grub/grub.cfg and /boot/efi/EFI/debian-10/grub.cfg came from !

Thus, here I am, at least able to use the system although forced to type those grub commands each time I start it up... And, since this is supposed to become my day to day laptop I'd really like to find a way to have grub immediately locating its config file, which I think must be achievable, specially since it is doable from the grub command line !
But, once again, I haven't been able to find a way to do it... until maybe one of you reading this brings me the so much expected solution ?

Thank you.

[p.H]

I've parsed all /dev/disk/by-id, zpool / zfs get guid and blkid output but couldn't determine where the 0d99d3724e04f532 value used in both /boot/grub/grub.cfg and /boot/efi/EFI/debian-10/grub.cfg came from !

0d99d3724e04f532 is the hexadecimal notation for decimal 980046881818473778. This is the first time I see a UUID with decimal notation.

Edit : did you compare the value of $root and $prefix before and after executing the search command ?

[tibdm]

Thank you p.H, I would not even have thought of converting from decimal to hexa !
That is already one thing that is clearer for me now.

Regarding the comparison of $root and $prefix, here is what can be observed:

Code: Select all

grub> echo $root
hd0,gpt3
grub> echo $prefix
(hd0,gpt3)/EFI/debian
grub> search --label --set root rpool
grub> echo $root
hd0,gpt1
grub> set prefix=($root)'/ROOT/debian-10@/boot/grub'
grub> echo $prefix
(hd0,gpt1)/ROOT/debian-10@/boot/grub

Do you have any clue on how to get further with this problem ?

Thanks

[p.H]

grub> echo $prefix
(hd0,gpt3)/EFI/debian

/EFI/debian, not /EFI/debian-10 ?
What about $cmdpath ? What is in /boot/efi/EFI/debian ?

[Head_on_a_Stick]

The ZFS configuration / installation was mainly carried out following the GitHub's zfsonlinux HOWTO for Debian Stretch https://github.com/zfsonlinux/zfs/wiki/ ... oot-on-ZFS.

Did you correct the `zpool create` command to take account of the newer ZFS version in buster, as suggested by that guide?

Caution: GRUB doesn't support all pool features from ZFS on Linux 0.7 or higher. If you install ZFS from backports or use Proxmox, add `-o feature@large_dnode=disabled` to the next command. Otherwise, GRUB will not find the ZFS pool.

FWIW, I installed a stretch system on a ZFS root using that guide and GRUB worked fine for me.

[tibdm]

@p.H

What is in /boot/efi/EFI/debian ?

You are right!
That is my problem...
Since I used grub-install with the --bootloader-id=debian-10 option, it created and populated the /boot/efi/EFI/debian-10 directory, not /boot/efi/EFI/debian...
I now simply renamed /boot/efi/EFI/debian-10 to /boot/efi/EFI/debian and grub finds its configuration file!

Although it seems to me an illogical thing then to be able to use the --bootloader-id= option with an arbitrary value if it beaks the grub loading process whenever this value is different than "debian"

Anyway, thank you very much for pointing me in the right direction and help me find a working solution!



@Head_on_a_Stick

Did you correct the `zpool create` command to take account of the newer ZFS version in buster, as suggested by that guide?

Yes I did... Kind of!
As a matter of fact, the statement made in the guide about GRUB not supporting the feature@large_dnode has now been replaced (the last edition of the guide was 11 hours ago ), what is now indicated is:

GRUB does not support all of the zpool features. See spa_feature_names in http://git.savannah.gnu.org/cgit/grub.g ... zfs.c#n276. This step creates a separate boot pool for /boot with the features limited to only those that GRUB supports, allowing the root pool to use any/all features. Note that GRUB opens the pool read-only, so all read-only compatible features are "supported" by GRUB.

In the configuration I set up, although I neither created a separate /boot pool, neither changed any default feature@ value while creating the rpool, everything seems to work flawlessly. And, to answer your question, yes, I tried with and without the -o feature@large_dnode=disabled, but didn't notice any difference.
Regarding GRUB's capabilities with ZFS filesystems, it might be of interest to read the following zfsonlinux thread, and specifically http://list.zfsonlinux.org/pipermail/zf ... 31505.html, where it is stated that GRUB's listing command (ls) is currently broken on ZFS but not its capability of loading initramfs / initrd...

Thanks however for the remark!

[tibdm]

PS: I would change the topic's title to [SOLVED]... if I only found out how to do it

[Dai_trying]

PS: I would change the topic's title to [SOLVED]... if I only found out how to do it

Edit the title of your first post.

[p.H]

I now simply renamed /boot/efi/EFI/debian-10 to /boot/efi/EFI/debian

Do you mean that /boot/efi/EFI/debian did not exist ?
Also, renaming /boot/efi/EFI/debian-10 should have broken the EFI boot entry Boot0000 which references the pathname \EFI\debian-10\shimx64.efi.

Although it seems to me an illogical thing then to be able to use the --bootloader-id= option with an arbitrary value if it breaks the grub loading process whenever this value is different than "debian"

Indeed. It was my understanding that any occurrence of the default value "debian" should be replaced with the value provided in --bootloader-id. By the way, does anyone know where this default value is taken from ? Is it related with GRUB_DISTRIBUTOR defined in /etc/default/grub ?

[tibdm]

Edit the title of your first post.

Thanks for the tip
Though, in regards to p.H last reply, I might wait a little before considering this topic as [SOLVED]...

[tibdm]

Do you mean that /boot/efi/EFI/debian did not exist ?
Also, renaming /boot/efi/EFI/debian-10 should have broken the EFI boot entry Boot0000 which references the pathname \EFI\debian-10\shimx64.efi.

Once again p.H you are perfectly right!

Nope, /boot/efi/EFI/debian did NOT exist...
I initially though of creating a symbolic link named /boot/efi/EFI/debian pointing to /boot/efi/EFI/debian-10, but this is not possible since the ESP is a FAT32 filesystem which doesn't support symlinks.
So I ended up copying the whole /boot/efi/EFI/debian-10 directory to /boot/efi/EFI/debian, so that they both existed in the ESP...

Now this is where things get weird:
Following your remark about the fact that removing the /boot/efi/EFI/debian-10 directory would break the boot entry Boot0000, I decided to give it a shot, removed the directory and restarted, obviously expecting the startup process to fail... What was my surprise in finding that the system restarted without problem!

My first reaction was to check the EFI boot entry and, oddly enough, found this:

Code: Select all

$ efibootmgr -v
BootCurrent: 0001
Timeout: 2 seconds
BootOrder: 0000,0001
Boot0000* debian-10   HD(3,GPT,581b4260-b5c1-4389-8e11-aa43f71d35b9,0x800,0x100000)/File(\EFI\DEBIAN-10\SHIMX64.EFI)
Boot0001* debian      HD(3,GPT,581b4260-b5c1-4389-8e11-aa43f71d35b9,0x800,0x100000)/File(\EFI\DEBIAN\GRUBX64.EFI)..BO

I'm puzzled !
The ONLY thing I did was duplicating /boot/efi/EFI/debian-10 to /boot/efi/EFI/debian and then restart, and BOOM, the EFI boot variables got re-written by themselves !?
I'm 100% positive I didn't interact with the EFI boot variables and notice that they are now ALL CAPS, while they initially used lower case. Also I observe ..BO at the end of the Boot0001 entry...
I really don't understand what is going on there, I've read quite a few pages about EFI booting while searching for a solution and before posting here, but I don't recall anything that could explain what I'm witnessing here.

Indeed. It was my understanding that any occurrence of the default value "debian" should be replaced with the value provided in --bootloader-id. By the way, does anyone know where this default value is taken from ? Is it related with GRUB_DISTRIBUTOR defined in /etc/default/grub ?

I'm glad to read that you share the logical idea to make use of the introduced --bootloader-id value when it is specified instead of the default one. And I'd also be interested in understanding where this default value is stored / copied from.

A this point, just to make sure that I got all the steps right and since I'm still experimenting with this system, I'll repeat the process. I'll wipe out the whole /boot directory and start over with the update-grub / grub-install commands. Restart, be stuck at the grub > prompt, etc... to finally duplicate the /boot/efi/EFI/debian-10 directory to /boot/efi/EFI/debian before a last restart.
I'll carefully observe the values returned by efibootmgr -v at each step and I'll come back here to confirm the result of the sequence.

In the meantime, any useful information that could be shared regarding what mechanism is involved here is more than welcome.

And thanks again p.H for sharing your remarks with me as they allow me, not only to start the system in a "normal" way, but also opens my mind to ways of exploring the problem that I wasn't able to find by myself.

[tibdm]

I'm back with the result of the experience I carried out regarding the GRUB install process for Debian Buster on a ZFS root.

What I did:

* Start the machine using a SystemRescueCD with ZFSonLinux USB Key (iso available @ http://list.zfsonlinux.org/zfs-iso/)

* Mount the rpool containing the Debian Buster system

Code: Select all

# mkdir /mnt/rpool
# zpool import -N -R /mnt/rpool rpool
# zfs mount rpool/ROOT/debian-10
# mount -t zfs rpool/ROOT/debian-10/var/log /mnt/rpool/var/log
# mount -t zfs rpool/ROOT/debian-10/var/tmp /mnt/rpool/var/tmp
# mount -t zfs rpool/tmp /mnt/rpool/tmp

* Zap the existing efi and grub data:

Code: Select all

# rm -rf mnt/rpool/boot/efi/*
# rm -rf mnt/rpool/boot/grub/*
# ls -l mnt/rpool/boot
total 32823
-rw-r--r-- 1 root root   206411 Jan 17 19:56 config-4.19.0-2-amd64
drwxr-xr-x 2 root root     4096 Jan  1  1970 efi
drwxr-xr-x 5 root root        7 Mar 20 01:39 grub
-rw-r--r-- 1 root root 27223819 Mar 18 08:49 initrd.img-4.19.0-2-amd64
-rw-r--r-- 1 root root  3345492 Jan 17 19:56 System.map-4.19.0-2-amd64
-rw-r--r-- 1 root root  5180704 Jan 17 19:56 vmlinuz-4.19.0-2-amd64
# ls -l mnt/rpool/boot/efi
total 0
# ls -l mnt/rpool/boot/grub
total 0

Since we booted from a USB key, the efi variables are mounted read-only, so we remount them with write access, then erase them:

Code: Select all

# mount -o remount,rw /sys/firmware/efi/efivars
# efibootmgr -v
BootCurrent: 0002
Timeout: 2 seconds
BootOrder: 0000,0001,0002
Boot0000* debian-10   HD(3,GPT,581b4260-b5c1-4389-8e11-aa43f71d35b9,0x800,0x100000)/File(\EFI\DEBIAN-10\SHIMX64.EFI)
Boot0001* debian      HD(3,GPT,581b4260-b5c1-4389-8e11-aa43f71d35b9,0x800,0x100000)/File(\EFI\DEBIAN\GRUBX64.EFI)..BO
Boot0002* UEFI: SanDisk, Partition 1      PciRoot(0x0)/Pci(0x14,0x0)/USB(0,0)/CDROM(1,0x12c,0x2d00)/HD(1,MBR,0x0,0x0,0xb40)..BO
# efibootmgr -b 0000 -B
BootCurrent: 0002
Timeout: 2 seconds
BootOrder: 0001,0002
Boot0001* debian
Boot0002* UEFI: SanDisk, Partition 1
# efibootmgr -b 0001 -B
BootCurrent: 0002
Timeout: 2 seconds
Boot0002* UEFI: SanDisk, Partition 1
# efibootmgr -b 0002 -B
BootCurrent: 0002
Timeout: 2 seconds
No BootOrder is set: firmware will attempt recovery

* CHROOT into rpool's Debian Buster environment:

Code: Select all

# mount --rbind /dev  /mnt/rpool/dev
# mount --rbind /proc /mnt/rpool/proc
# mount --rbind /sys  /mnt/rpool/sys
# chroot /mnt/rpool /bin/bash --login

* Perform update-grub, then check EFI boot entries and /boot directory:

Code: Select all

# update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.19.0-2-amd64
Found initrd image: /boot/initrd.img-4.19.0-2-amd64
Adding boot menu entry for EFI firmware configuration
done
# efibootmgr -v
BootCurrent: 0002
Timeout: 2 seconds
No BootOrder is set: firmware will attempt recovery
# ls -l /boot/grub
total 5
-r--r--r-- 1 root root  4818 Mar 20 01:23 grub.cfg
# ls -l /boot/efi
total 0

At this stage we see that:

• /boot/grub/grub.cfg is created
• EFI boot entries are untouched

* Perform grub-install, then check EFI boot entries and /boot directory:

Code: Select all

# grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=debian-10 --recheck --no-floppy
Installing for x86_64-efi platform.
Installation finished. No error reported.
# efibootmgr -v
BootCurrent: 0002
Timeout: 2 seconds
BootOrder: 0000
Boot0000* debian-10   HD(3,GPT,581b4260-b5c1-4389-8e11-aa43f71d35b9,0x800,0x100000)/File(\EFI\debian-10\shimx64.efi)
# ls -l /boot/grub
total 23
drwxr-xr-x 2 root root    3 Mar 20 01:39 fonts
-r--r--r-- 1 root root 4818 Mar 20 01:38 grub.cfg
-rw-r--r-- 1 root root 1024 Mar 20 01:39 grubenv
drwxr-xr-x 2 root root   40 Mar 20 01:39 locale
drwxr-xr-x 2 root root  270 Mar 20 01:39 x86_64-efi
# ls -l /boot/efi
total 4
drwxr-xr-x 3 root root 4096 Mar 20 01:39 EFI
# ls -l /boot/efi/EFI/
total 4
drwxr-xr-x 2 root root 4096 Mar 20 01:39 debian-10
# ls -l /boot/efi/EFI/debian-10
total 3768
-rwxr-xr-x 1 root root   72144 Mar 20 01:39 fbx64.efi
-rwxr-xr-x 1 root root     113 Mar 20 01:39 grub.cfg
-rwxr-xr-x 1 root root 1435040 Mar 20 01:39 grubx64.efi
-rwxr-xr-x 1 root root 1168464 Mar 20 01:39 mmx64.efi
-rwxr-xr-x 1 root root 1169528 Mar 20 01:39 shimx64.efi

At this stage we see that:

• /boot/grub/ gets fully populated
• /boot/efi/EFI/debian-10 gets created and fully populated
• A new EFI boot entry is created, that targets \EFI\debian-10\shimx64.efi

* Exit the CHROOTED environment, unmount everything and reboot:

Code: Select all

# exit
# mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | xargs -i{} umount -lf {}
# zpool export rpool
# reboot

* Reach the GRUB command line:
As previously, at this stage, the boot process is immediately landing on the grub command line. We grab this opportunity to inspect some variables values:

Code: Select all

grub> echo $root
hd0,gpt3
grub> echo $prefix
(hd0,gpt3)/EFI/debian
grub> echo $cmdpath
(hd0,gpt3)/EFI/debian-10

It is interesting to note here that the $prefix value is set to (hd0,gpt3)/EFI/debian, which seems wrong, while the $cmdpath value is (hd0,gpt3)/EFI/debian-10 which seems right...

We issue the following grub command to reach the standard GRUB menu and load the system:

Code: Select all

grub> configfile $cmdpath/grub.cfg

* Inspect the current EFI boot entries:

Code: Select all

# efibootmgr -v
BootCurrent: 0000
Timeout: 2 seconds
BootOrder: 0000
Boot0000* debian-10   HD(3,GPT,581b4260-b5c1-4389-8e11-aa43f71d35b9,0x800,0x100000)/File(\EFI\debian-10\shimx64.efi)

* Duplicate /boot/efi/EFI/debian-10 to /boot/efi/EFI/debian, then reboot:

Code: Select all

# cp -a /boot/efi/EFI/debian-10 /boot/efi/EFI/debian
# ls -l /boot/efi/EFI/
total 8
drwxr-xr-x 2 root root 4096 Mar 20 01:39 debian
drwxr-xr-x 2 root root 4096 Mar 20 01:39 debian-10
# reboot

At this point the system loads without a problem...

* Perform a last inspection of the EFI boot entries:

Code: Select all

# efibootmgr -v
BootCurrent: 0000
Timeout: 2 seconds
BootOrder: 0000,0001
Boot0000* debian-10   HD(3,GPT,581b4260-b5c1-4389-8e11-aa43f71d35b9,0x800,0x100000)/File(\EFI\DEBIAN-10\SHIMX64.EFI)
Boot0001* debian      HD(3,GPT,581b4260-b5c1-4389-8e11-aa43f71d35b9,0x800,0x100000)/File(\EFI\DEBIAN\GRUBX64.EFI)..BO

We find that the firmware has re-written the entries, but apparently still uses Boot0000.


My understanding of all of the above is that:

• the grub-install command wrongly sets the prefix variable when the --bootloader-id option is given an arbitrary value
• although the initial Boot0000 EFI parameter is activated at startup, it runs the code from the duplicated /boot/efi/EFI/debian directory
• this means that the current implementation of grub-efi-amd64 (2.02+dfsg1-12) present in Debian Buster (frozen 2019-03-11) has a bug

If this conclusion is correct, what should be done?
This is far beyond my competence and I do not know what to do if it is not to introduce a bug report to the Debian development team, which I have never done and ignore how to do...

Any indication would be greatly appreciated.

Thank you.

[p.H]

I did a bit of investigation.

1) My tests confirmed that the default --bootloader-id value is taken from GRUB_DISTRIBUTOR defined in /etc/default/grub. So it may be more convenient to set permanently the value in this file instead of passing it in the command line.

2) I think I found why $prefix ignores --bootloader-id and GRUB_DISTRIBUTOR.
Usually, GRUB's core image is built by grub-mkimage and embeds the $prefix value or a minimal config file used to set the $prefix value.
However when installing GRUB for EFI secure boot, grub-install seems to use a pre-built signed core image /usr/lib/grub/x86_64-efi-signed/grubx64.efi.signed which has the prefix hardcoded with "debian". This image also seems to include most modules, which also explains why GRUB starts in normal mode instead of rescue mode as expected when the $prefix location cannot be found. I guess the signed image cannot be modified because the signature would not match any more. So the prebuilt signed image works only with the default distributor/bootloader ID value "Debian". IMO, an easy fix would be to use $cmdpath as the value for $prefix instead of a hardcoded value.

By default, GRUB is installed for EFI secure boot if the package grub-efi-amd64-signed is installed unless --no-uefi-secure-boot is passed. grub-efi-amd64-signed is a Recommends: dependency of grub-efi-amd64-bin, so it is installed by default. If you do not need secure boot, you can either remove grub-efi-amd64-signed or pass --no-uefi-secure-boot when running grub-install.

although the initial Boot0000 EFI parameter is activated at startup, it runs the code from the duplicated /boot/efi/EFI/debian directory

I disagree. It still runs the executable located at the Boot0000 pathname, but this program uses /boot/efi/EFI/debian/grub.cfg which now exist. I think it would work the same if you only copy or move /boot/efi/EFI/debian-10/grub.cfg into /boot/efi/EFI/debian.

I have not investigated the second EFI boot entry issue yet.

[tibdm]

@p.H

First of all, thank you for having thoroughly studied this question and given such detailed information on exactly what is happening in the described situation. It really goes far beyond what I would have been able to understand by myself!

I disagree. It still runs the executable located at the Boot0000 pathname, but this program uses /boot/efi/EFI/debian/grub.cfg which now exist. I think it would work the same if you only copy or move /boot/efi/EFI/debian-10/grub.cfg into /boot/efi/EFI/debian.

Once again here you are perfectly right, I've now removed all /boot/efi/EFI/debian/*.efi files and the boot process goes on perfectly well.

IMO, an easy fix would be to use $cmdpath as the value for $prefix instead of a hardcoded value

As I said before, my understanding of the involved processes is far from yours, so my question here on that point would be: Is this something that can be achieved as a system user or does it require some grub code modification?

I have not investigated the second EFI boot entry issue yet.

Regarding this last point, I must say that the observed behavior is rather surprising. When the initial /boot/efi/EFI/debian-10 directory is duplicated to /boot/efi/EFI/debian, a new EFI boot entry is created on next boot. This new EFI boot entry line ends with the ..BO characters, as is the case when bootable USB Keys are present at boot time, they also generate an EFI boot entry ending with ..BO.

AS one might expect, the EFI boot entry only gets created when the /boot/efi/EFI/debian directory contains *.efi files.

Now what is more surprising is that, contrarily to what happens when a USB Key is removed, if the /boot/efi/EFI/debian directory is deleted or renamed, the EFI boot entry is nor removed nor modified. Also, the boot entry only gets created when the duplicated directory gets named as "debian", any other name will not generate it automatically.

I currently only keep the grub.cfg file in /boot/efi/EFI/debian, which allows for the system to startup properly and doesn't create the second EFI boot record...

Thanks again for having shed light on all of this.

[p.H]

Is this something that can be achieved as a system user or does it require some grub code modification?

It requires to modify the signed GRUB core image. This image is signed by Debian, so only Debian can modify it. I think I will submit a bug report against grub-efi-amd64-signed.

A maybe cleaner workaround than copying grub.cfg into /boot/efi/EFI/debian would be to create an additional /boot/efi/EFI/debian/grub.cfg file with the following contents :

Code: Select all

configfile $cmdpath/grub.cfg

so that in the end GRUB would use the grub.cfg file present in its own directory. This allows multiple instances of GRUB using different grub.cfg files to load different systems.

But is secure boot enforced on the machine ? If not, you can just install GRUB without secure boot support. This way the $prefix value pointing to /boot/grub will be embedded directly in GRUB's core image and it won't need to use an initial grub.cfg file in the EFI partition.

2) I did preliminary testing regarding the EFI boot entry, trying to follow your steps as closely as possible but the second variable was not created.

[tibdm]

I think I will submit a bug report against grub-efi-amd64-signed

Great! Nice to know that you'll raise the question to the Debian team since you're certainly more able than me to provide them with the necessary information in an adequate way, thank you.

I did preliminary testing regarding the EFI boot entry, trying to follow your steps as closely as possible but the second variable was not created.

This is strange, all I can tell you is that I've repeated the exercise many times and that, on my system, it always happens as described...

Once again, I want to thank you p.H for sharing your knowledge and taking the time to give a clear explanation of what was going and why it was happening.
I'll now mark this thread as [SOLVED] since you brought a working solution to the problem, please feel free to give here the reference to the bug tracking issue you'll introduce so anyone interested in the management of this GRUB shortcoming in Debian (like myself) will be ale to follow the progress on the matter.

Thanks a lot.

[p.H]

https://bugs.debian.org/cgi-bin/bugrepo ... bug=925309