vm usb passtrought won't work after kernel upgrade [solved]

Kernels & Hardware, configuring network, installing services

vm usb passtrought won't work after kernel upgrade [solved]

Postby daggs » 2018-11-03 05:16

Greetings,

I have a Debian stretch installation version 9.5, my router is a vm on that machine that has one usb device (wireless) configured as passthough.
the kernel version was 4.15.11-1~bpo9+1 (installed due to usb3 issues), the relevant vm config is as follows:
Code: Select all
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x148f'/>
        <product id='0x5572'/>
      </source>
      <address type='usb' bus='0' port='1'/>
    </hostdev>

the address part was added auto by virsh.
this system uses another usb dongle (dvb) which resulted in usb -110 errors. I've searched for the reason and found out that in kernel 4.9 some memory apis were changed and such errors are results of memory corruption which will be fixed in later versions.
I've decided to upgrade the kernel to 4.18.6-1~bpo9+1 which I did. since then, the vm won't boot with the usb passthough set, the error I'm getting is this:
Code: Select all
error: Failed to start domain router
error: internal error: qemu unexpectedly closed the monitor: 2018-11-02T13:43:32.174730Z qemu-system-x86_64: -device usb-host,hostbus=1,hostaddr=4,id=hostdev4,bus=usb.0,port=1: failed to find host usb device 1:4

the output of lsusb -t is this:
Code: Select all
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 480M
    |__ Port 1: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 12M
    |__ Port 1: Dev 2, If 1, Class=Human Interface Device, Driver=usbhid, 12M
    |__ Port 2: Dev 3, If 0, Class=Vendor Specific Class, Driver=dvb_usb_cxusb, 480M
    |__ Port 6: Dev 4, If 0, Class=Vendor Specific Class, Driver=rt2800usb, 480M
    |__ Port 8: Dev 5, If 0, Class=Mass Storage, Driver=usb-storage, 480M

rt2800usb is the device I want pass to the vm.

any ideas how I can fix this?

thanks,

Dagg.
Last edited by daggs on 2018-11-09 11:37, edited 1 time in total.
daggs
 
Posts: 14
Joined: 2018-04-21 16:12

Re: vm with usb passtrought won't work after kernel upgrade

Postby CwF » 2018-11-04 14:07

Looks like you are passing the device and not the port?

Try passing the port with vfio.

Edited example:
Code: Select all
# lspci -v -d 1033:
07:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04) (prog-if 30 [XHCI])
   Subsystem: Super Micro Computer Inc uPD720200 USB 3.0 Host Controller
   ......
   Kernel driver in use: vfio-pci
   Kernel modules: xhci_pci

08:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04) (prog-if 30 [XHCI])
   Subsystem: Super Micro Computer Inc uPD720200 USB 3.0 Host Controller
   ......
   Kernel driver in use: xhci_hcd
   Kernel modules: xhci_pci

Here there are two usb3 controllers, one is for host use, the first one is attached to the vm (vfio-pci)
. When you pass only the usb device there is a dependence on how it is handled on the host, ie how will the driver release it. If you pass the whole port it simply yanks it from the host, the vm gets the native port with whatever devices Some change in the kernel may be preventing the dongle from releasing from the host, I've had this issue.

I'm also interested in what is running in this vm!
CwF
 
Posts: 168
Joined: 2018-06-20 15:16

Re: vm with usb passtrought won't work after kernel upgrade

Postby daggs » 2018-11-04 16:02

CwF wrote:Looks like you are passing the device and not the port?

Try passing the port with vfio.

Edited example:
Code: Select all
# lspci -v -d 1033:
07:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04) (prog-if 30 [XHCI])
   Subsystem: Super Micro Computer Inc uPD720200 USB 3.0 Host Controller
   ......
   Kernel driver in use: vfio-pci
   Kernel modules: xhci_pci

08:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04) (prog-if 30 [XHCI])
   Subsystem: Super Micro Computer Inc uPD720200 USB 3.0 Host Controller
   ......
   Kernel driver in use: xhci_hcd
   Kernel modules: xhci_pci

Here there are two usb3 controllers, one is for host use, the first one is attached to the vm (vfio-pci)
. When you pass only the usb device there is a dependence on how it is handled on the host, ie how will the driver release it. If you pass the whole port it simply yanks it from the host, the vm gets the native port with whatever devices Some change in the kernel may be preventing the dongle from releasing from the host, I've had this issue.

can you show an example of the xml? afaik, vfio-pci is used to pass entire pci devices, I have only one usb crtl. infact I use this method to pass all my 4 nics to te vm.
how to do separate specific ports that way?
CwF wrote:I'm also interested in what is running in this vm!

my router is on that vm.
daggs
 
Posts: 14
Joined: 2018-04-21 16:12

Re: vm with usb passtrought won't work after kernel upgrade

Postby CwF » 2018-11-04 16:17

daggs wrote:
CwF wrote:afaik, vfio-pci is used to pass entire pci devices, I have only one usb crtl. infact I use this method to pass all my 4 nics to te vm.
how to do separate specific ports that way?

Correct, if you do only have one your screwed I guess.
Try this with no vm's running
Code: Select all
for usb_ctrl in $(find /sys/bus/usb/devices/usb* -maxdepth 0 -type l); do pci_path="$(dirname "$(realpath "${usb_ctrl}")")"; echo "Bus $(cat "${usb_ctrl}/busnum") --> $(basename $pci_path) (IOMMU group $(basename $(realpath $pci_path/iommu_group)))"; lsusb -s "$(cat "${usb_ctrl}/busnum"):"; echo; done

post the response and we'll go from there. We're looking for how the usb topology is grouped. If there is a more than one group we may be able to steal one for the vm.
CwF
 
Posts: 168
Joined: 2018-06-20 15:16

Re: vm with usb passtrought won't work after kernel upgrade

Postby daggs » 2018-11-04 16:26

CwF wrote:
daggs wrote:
CwF wrote:afaik, vfio-pci is used to pass entire pci devices, I have only one usb crtl. infact I use this method to pass all my 4 nics to te vm.
how to do separate specific ports that way?

Correct, if you do only have one your screwed I guess.
Try this with no vm's running
Code: Select all
for usb_ctrl in $(find /sys/bus/usb/devices/usb* -maxdepth 0 -type l); do pci_path="$(dirname "$(realpath "${usb_ctrl}")")"; echo "Bus $(cat "${usb_ctrl}/busnum") --> $(basename $pci_path) (IOMMU group $(basename $(realpath $pci_path/iommu_group)))"; lsusb -s "$(cat "${usb_ctrl}/busnum"):"; echo; done

post the response and we'll go from there. We're looking for how the usb topology is grouped. If there is a more than one group we may be able to steal one for the vm.

it is a bit problematic to stop the vm.
I'm not passing any other usb to the vm so I assume it is safe to run the cmd:
Code: Select all
Bus 1 --> 0000:00:14.0 (IOMMU group 3)
Bus 001 Device 005: ID 0781:5583 SanDisk Corp.
Bus 001 Device 004: ID 148f:5572 Ralink Technology, Corp. RT5572 Wireless Adapter
Bus 001 Device 010: ID 0572:c688 Conexant Systems (Rockwell), Inc. Geniatech T230 DVB-T2 TV Stick
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 2 --> 0000:00:14.0 (IOMMU group 3)
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

daggs
 
Posts: 14
Joined: 2018-04-21 16:12

Re: vm with usb passtrought won't work after kernel upgrade

Postby CwF » 2018-11-04 16:40

...totally safe. If your passing one to a vm then its listing, it's bus, won't be listed. So if that is all you have, 2 buses in group 3 then you cannot pass the port itself. Sorry, it likely would solve the issue but without a suitable onboard port the only option would be an add in card.
CwF
 
Posts: 168
Joined: 2018-06-20 15:16

Re: vm with usb passtrought won't work after kernel upgrade

Postby daggs » 2018-11-04 16:56

CwF wrote:...totally safe. If your passing one to a vm then its listing, it's bus, won't be listed. So if that is all you have, 2 buses in group 3 then you cannot pass the port itself. Sorry, it likely would solve the issue but without a suitable onboard port the only option would be an add in card.

in that case, how does kernel change causes this?
daggs
 
Posts: 14
Joined: 2018-04-21 16:12

Re: vm with usb passtrought won't work after kernel upgrade

Postby CwF » 2018-11-04 17:17

I don't know specifically. I've just found differences in how certain things are handled, in this case released from the host. Sometimes the problem is re-initializing it on the host after vm use. I haven't passed a dongle lately but have seen variation in bluetooth, wifi, tv, and other types. In all cases passing the port works, so I pass a port. For example I used to 'device' pass a logi unifying dongle also used by the host, that doesn't always work.

So your 4 eth ports, is that a 4 port card and if so do you pass all 4, or 2 pairs, or singly?
CwF
 
Posts: 168
Joined: 2018-06-20 15:16

Re: vm with usb passtrought won't work after kernel upgrade

Postby daggs » 2018-11-04 17:59

CwF wrote:I don't know specifically. I've just found differences in how certain things are handled, in this case released from the host. Sometimes the problem is re-initializing it on the host after vm use. I haven't passed a dongle lately but have seen variation in bluetooth, wifi, tv, and other types. In all cases passing the port works, so I pass a port. For example I used to 'device' pass a logi unifying dongle also used by the host, that doesn't always work.

So your 4 eth ports, is that a 4 port card and if so do you pass all 4, or 2 pairs, or singly?

4 cards, one port each, passing the pci lane.
my issue happens on boot, not after sometime
daggs
 
Posts: 14
Joined: 2018-04-21 16:12

Re: vm with usb passtrought won't work after kernel upgrade

Postby CwF » 2018-11-04 18:37

By 'sometimes' I don't mean sporadic on one machine, but works here in one machines configuration and does not work in another with differing kernel and in the case of the unifying receiver, the package solaar I think it was.
my issue happens on boot, not after sometime

You mean when you boot the vm, yes? So the usb is pre-defined to pass, the check if it is available at boot causes the vm to fail. What if you pass it after the vm is up and running without it? I believe this would test if can release from the host. I have two things that worked prior to 4.4x. IIRC I had a tv dongle that would pass under 3.x that now requires the port be passed. Don't know? At that time it was pci-stub and that maybe didn't work until vfio was in 3.9 or something?

I suppose you could try to blacklist or unload anything on the host. Look for something new that has change, perhaps before the host had no driver for the device and now it does?

I ask on the eth since I'm still looking for a 4 port card that can divy up 4 ports, mine does two pairs... I also use a dual port that can divide them, point being you need another slot for a usb card!?
CwF
 
Posts: 168
Joined: 2018-06-20 15:16

Re: vm with usb passtrought won't work after kernel upgrade

Postby daggs » 2018-11-04 19:17

CwF wrote:By 'sometimes' I don't mean sporadic on one machine, but works here in one machines configuration and does not work in another with differing kernel and in the case of the unifying receiver, the package solaar I think it was.
my issue happens on boot, not after sometime

You mean when you boot the vm, yes? So the usb is pre-defined to pass, the check if it is available at boot causes the vm to fail. What if you pass it after the vm is up and running without it? I believe this would test if can release from the host. I have two things that worked prior to 4.4x. IIRC I had a tv dongle that would pass under 3.x that now requires the port be passed. Don't know? At that time it was pci-stub and that maybe didn't work until vfio was in 3.9 or something?

I suppose you could try to blacklist or unload anything on the host. Look for something new that has change, perhaps before the host had no driver for the device and now it does?

I ask on the eth since I'm still looking for a 4 port card that can divy up 4 ports, mine does two pairs... I also use a dual port that can divide them, point being you need another slot for a usb card!?

has none. all the pci slots are taken.
I think I'll take this question to the libvirt ml, see what they have to say.

thanks for the help.
daggs
 
Posts: 14
Joined: 2018-04-21 16:12

Re: vm with usb passtrought won't work after kernel upgrade

Postby CwF » 2018-11-04 19:39

daggs wrote:has none. all the pci slots are taken.
I think I'll take this question to the libvirt ml, see what they have to say.

Yes, I see, hence the suggestion of combining 4 cards into 2 or 1, leaving room for a usb card. Future note; not all usb cards vfio pass well, its own subject.

Please post back if there is another solution. Good luck.
CwF
 
Posts: 168
Joined: 2018-06-20 15:16

Re: vm with usb passtrought won't work after kernel upgrade

Postby daggs » 2018-11-09 11:37

Solved!

craparmor was to blame, I've disabled it in kernel params and all works again.
daggs
 
Posts: 14
Joined: 2018-04-21 16:12


Return to System configuration

Who is online

Users browsing this forum: No registered users and 4 guests

fashionable