How to dual boot as vm and bare metal

[sarek1924]

I have a debian partition on my SSD that boots fine both under the MB's BIOS and under bhyve (freebsd hypervisor) after installing nvidia-driver the machine only boots on bare metal not bhyve (bhyve has known issues with nvidia passthrough). Any ideas on how to tell it to ignore the fact that there are no active GPU's and either not load the module or load it with no devices?

Note I am brand new to debian but have decades of experience with FreeBSD and frankly the only reason I am even trying debian is CUDA just does not work on FreeBSD as it currently stands.

[CwF]

This is the issue with having an absolute xorg.conf file. Nouveau doesn't have this problem btw. You either need to script some detection or manually comment out the offending declarations. If you are dumped to a tty, you can log on as root and use the obtuse editor of choice to edit these config files, then on logoff xorg will reread these files and load your dm, or execute other logon methods.

[kent_dorfman766]

It's no easy task to set up a more sophisiticated OS image to work transparently under a hypervisor and to boot natively. Obviously, as you've found out, the hardware presented is different, depending upon the configuration. The modern nvidia drivers are loaded at boot time in the initrd. You will need to remove the driver from the initrd image, and blacklist it from being automatically loaded otherwise.

You'll need to use a detection script to manually load the nvidia driver using insmod for the cases where you boot natively. Once the driver is loaded then the display manager can function properly. For the VM instance, I'll assume you're just using the qxl driver if any gui at all.

CUDA support is a roadblocker for me as well. I ended up running BSD as a VM under qemu/kvm in Debian 12 (soon to move to stable).

[CwF]

You will need to remove the driver from the initrd image, and blacklist it from being automatically loaded otherwise.

I don't think that is required. If no hardware is present while configured for nvidia, it doesn't load anything, I have vm'ified a few such nvidia broken disk images and with absent hardware and they fall back and land at the prompt, though you might have to actually invoke the tty with a Ctrl+Alt+F*. From there delete all xorg declarations, logoff, logon, startx. The vga/virtio/qxl/vmware/gallium xorg driver needs no config, just make sure the packages are installed, along with qemu-agent and the like...

Blacklisting is often done and almost never required, and shouldn't be here. If anything there is a nouveau blacklist from nvidia. Xorgs temptation to load a driver depends on it being the most obvious first and available choice - unless specified. So simply eliminate the declarations and xorg will find the obvious choice to be qxl, or whatever, and use it.

Thinking about it the quickest solution is to plan ahead, set up for vm with no xorg.conf, and move the nvidia xorg into place while in the vm for the next boot to be intentionally bare metal. Then move the file out of location before shutdown, intending the next boot as a vm.

On more thought, why a vm at all without the hardware? If you do pass the gpu take note if the xorg file includes a specific address for the gpu, that will be different between modes.

[sarek1924]

I should mentioned that I do not want X on the machine I want to use it as a pure GPU computing engine (pytorch).... so I don't care if it is onlty text more on bare metal.. the idea is I write the non-AI part of my app on the VM and test it on the bare metal

[CwF]

I do not want X on the machine

Then just the middle part...

I have vm'ified a few such nvidia broken disk images and with absent hardware and they fall back and land at the prompt, though you might have to actually invoke the tty with a Ctrl+Alt+F*

I don't know why it would not work unaltered.