Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

vm usb passtrought won't work after kernel upgrade [solved]

Linux Kernel, Network, and Services configuration.
Post Reply
Message
Author
daggs
Posts: 17
Joined: 2018-04-21 16:12

vm usb passtrought won't work after kernel upgrade [solved]

#1 Post by daggs »

Greetings,

I have a Debian stretch installation version 9.5, my router is a vm on that machine that has one usb device (wireless) configured as passthough.
the kernel version was 4.15.11-1~bpo9+1 (installed due to usb3 issues), the relevant vm config is as follows:

Code: Select all

    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x148f'/>
        <product id='0x5572'/>
      </source>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
the address part was added auto by virsh.
this system uses another usb dongle (dvb) which resulted in usb -110 errors. I've searched for the reason and found out that in kernel 4.9 some memory apis were changed and such errors are results of memory corruption which will be fixed in later versions.
I've decided to upgrade the kernel to 4.18.6-1~bpo9+1 which I did. since then, the vm won't boot with the usb passthough set, the error I'm getting is this:

Code: Select all

error: Failed to start domain router
error: internal error: qemu unexpectedly closed the monitor: 2018-11-02T13:43:32.174730Z qemu-system-x86_64: -device usb-host,hostbus=1,hostaddr=4,id=hostdev4,bus=usb.0,port=1: failed to find host usb device 1:4
the output of lsusb -t is this:

Code: Select all

/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 480M
    |__ Port 1: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 12M
    |__ Port 1: Dev 2, If 1, Class=Human Interface Device, Driver=usbhid, 12M
    |__ Port 2: Dev 3, If 0, Class=Vendor Specific Class, Driver=dvb_usb_cxusb, 480M
    |__ Port 6: Dev 4, If 0, Class=Vendor Specific Class, Driver=rt2800usb, 480M
    |__ Port 8: Dev 5, If 0, Class=Mass Storage, Driver=usb-storage, 480M
rt2800usb is the device I want pass to the vm.

any ideas how I can fix this?

thanks,

Dagg.
Last edited by daggs on 2018-11-09 11:37, edited 1 time in total.

CwF
Global Moderator
Global Moderator
Posts: 2638
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 192 times

Re: vm with usb passtrought won't work after kernel upgrade

#2 Post by CwF »

Looks like you are passing the device and not the port?

Try passing the port with vfio.

Edited example:

Code: Select all

# lspci -v -d 1033:
07:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04) (prog-if 30 [XHCI])
	Subsystem: Super Micro Computer Inc uPD720200 USB 3.0 Host Controller
	......
	Kernel driver in use: vfio-pci
	Kernel modules: xhci_pci

08:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04) (prog-if 30 [XHCI])
	Subsystem: Super Micro Computer Inc uPD720200 USB 3.0 Host Controller
	......
	Kernel driver in use: xhci_hcd
	Kernel modules: xhci_pci
Here there are two usb3 controllers, one is for host use, the first one is attached to the vm (vfio-pci)
. When you pass only the usb device there is a dependence on how it is handled on the host, ie how will the driver release it. If you pass the whole port it simply yanks it from the host, the vm gets the native port with whatever devices Some change in the kernel may be preventing the dongle from releasing from the host, I've had this issue.

I'm also interested in what is running in this vm!

daggs
Posts: 17
Joined: 2018-04-21 16:12

Re: vm with usb passtrought won't work after kernel upgrade

#3 Post by daggs »

CwF wrote:Looks like you are passing the device and not the port?

Try passing the port with vfio.

Edited example:

Code: Select all

# lspci -v -d 1033:
07:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04) (prog-if 30 [XHCI])
	Subsystem: Super Micro Computer Inc uPD720200 USB 3.0 Host Controller
	......
	Kernel driver in use: vfio-pci
	Kernel modules: xhci_pci

08:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04) (prog-if 30 [XHCI])
	Subsystem: Super Micro Computer Inc uPD720200 USB 3.0 Host Controller
	......
	Kernel driver in use: xhci_hcd
	Kernel modules: xhci_pci
Here there are two usb3 controllers, one is for host use, the first one is attached to the vm (vfio-pci)
. When you pass only the usb device there is a dependence on how it is handled on the host, ie how will the driver release it. If you pass the whole port it simply yanks it from the host, the vm gets the native port with whatever devices Some change in the kernel may be preventing the dongle from releasing from the host, I've had this issue.
can you show an example of the xml? afaik, vfio-pci is used to pass entire pci devices, I have only one usb crtl. infact I use this method to pass all my 4 nics to te vm.
how to do separate specific ports that way?
CwF wrote: I'm also interested in what is running in this vm!
my router is on that vm.

CwF
Global Moderator
Global Moderator
Posts: 2638
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 192 times

Re: vm with usb passtrought won't work after kernel upgrade

#4 Post by CwF »

daggs wrote:
CwF wrote:afaik, vfio-pci is used to pass entire pci devices, I have only one usb crtl. infact I use this method to pass all my 4 nics to te vm.
how to do separate specific ports that way?
Correct, if you do only have one your screwed I guess.
Try this with no vm's running

Code: Select all

for usb_ctrl in $(find /sys/bus/usb/devices/usb* -maxdepth 0 -type l); do pci_path="$(dirname "$(realpath "${usb_ctrl}")")"; echo "Bus $(cat "${usb_ctrl}/busnum") --> $(basename $pci_path) (IOMMU group $(basename $(realpath $pci_path/iommu_group)))"; lsusb -s "$(cat "${usb_ctrl}/busnum"):"; echo; done
post the response and we'll go from there. We're looking for how the usb topology is grouped. If there is a more than one group we may be able to steal one for the vm.

daggs
Posts: 17
Joined: 2018-04-21 16:12

Re: vm with usb passtrought won't work after kernel upgrade

#5 Post by daggs »

CwF wrote:
daggs wrote:
CwF wrote:afaik, vfio-pci is used to pass entire pci devices, I have only one usb crtl. infact I use this method to pass all my 4 nics to te vm.
how to do separate specific ports that way?
Correct, if you do only have one your screwed I guess.
Try this with no vm's running

Code: Select all

for usb_ctrl in $(find /sys/bus/usb/devices/usb* -maxdepth 0 -type l); do pci_path="$(dirname "$(realpath "${usb_ctrl}")")"; echo "Bus $(cat "${usb_ctrl}/busnum") --> $(basename $pci_path) (IOMMU group $(basename $(realpath $pci_path/iommu_group)))"; lsusb -s "$(cat "${usb_ctrl}/busnum"):"; echo; done
post the response and we'll go from there. We're looking for how the usb topology is grouped. If there is a more than one group we may be able to steal one for the vm.
it is a bit problematic to stop the vm.
I'm not passing any other usb to the vm so I assume it is safe to run the cmd:

Code: Select all

Bus 1 --> 0000:00:14.0 (IOMMU group 3)
Bus 001 Device 005: ID 0781:5583 SanDisk Corp. 
Bus 001 Device 004: ID 148f:5572 Ralink Technology, Corp. RT5572 Wireless Adapter
Bus 001 Device 010: ID 0572:c688 Conexant Systems (Rockwell), Inc. Geniatech T230 DVB-T2 TV Stick
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 2 --> 0000:00:14.0 (IOMMU group 3)
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub


CwF
Global Moderator
Global Moderator
Posts: 2638
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 192 times

Re: vm with usb passtrought won't work after kernel upgrade

#6 Post by CwF »

...totally safe. If your passing one to a vm then its listing, it's bus, won't be listed. So if that is all you have, 2 buses in group 3 then you cannot pass the port itself. Sorry, it likely would solve the issue but without a suitable onboard port the only option would be an add in card.

daggs
Posts: 17
Joined: 2018-04-21 16:12

Re: vm with usb passtrought won't work after kernel upgrade

#7 Post by daggs »

CwF wrote:...totally safe. If your passing one to a vm then its listing, it's bus, won't be listed. So if that is all you have, 2 buses in group 3 then you cannot pass the port itself. Sorry, it likely would solve the issue but without a suitable onboard port the only option would be an add in card.
in that case, how does kernel change causes this?

CwF
Global Moderator
Global Moderator
Posts: 2638
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 192 times

Re: vm with usb passtrought won't work after kernel upgrade

#8 Post by CwF »

I don't know specifically. I've just found differences in how certain things are handled, in this case released from the host. Sometimes the problem is re-initializing it on the host after vm use. I haven't passed a dongle lately but have seen variation in bluetooth, wifi, tv, and other types. In all cases passing the port works, so I pass a port. For example I used to 'device' pass a logi unifying dongle also used by the host, that doesn't always work.

So your 4 eth ports, is that a 4 port card and if so do you pass all 4, or 2 pairs, or singly?

daggs
Posts: 17
Joined: 2018-04-21 16:12

Re: vm with usb passtrought won't work after kernel upgrade

#9 Post by daggs »

CwF wrote:I don't know specifically. I've just found differences in how certain things are handled, in this case released from the host. Sometimes the problem is re-initializing it on the host after vm use. I haven't passed a dongle lately but have seen variation in bluetooth, wifi, tv, and other types. In all cases passing the port works, so I pass a port. For example I used to 'device' pass a logi unifying dongle also used by the host, that doesn't always work.

So your 4 eth ports, is that a 4 port card and if so do you pass all 4, or 2 pairs, or singly?
4 cards, one port each, passing the pci lane.
my issue happens on boot, not after sometime

CwF
Global Moderator
Global Moderator
Posts: 2638
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 192 times

Re: vm with usb passtrought won't work after kernel upgrade

#10 Post by CwF »

By 'sometimes' I don't mean sporadic on one machine, but works here in one machines configuration and does not work in another with differing kernel and in the case of the unifying receiver, the package solaar I think it was.
my issue happens on boot, not after sometime
You mean when you boot the vm, yes? So the usb is pre-defined to pass, the check if it is available at boot causes the vm to fail. What if you pass it after the vm is up and running without it? I believe this would test if can release from the host. I have two things that worked prior to 4.4x. IIRC I had a tv dongle that would pass under 3.x that now requires the port be passed. Don't know? At that time it was pci-stub and that maybe didn't work until vfio was in 3.9 or something?

I suppose you could try to blacklist or unload anything on the host. Look for something new that has change, perhaps before the host had no driver for the device and now it does?

I ask on the eth since I'm still looking for a 4 port card that can divy up 4 ports, mine does two pairs... I also use a dual port that can divide them, point being you need another slot for a usb card!?

daggs
Posts: 17
Joined: 2018-04-21 16:12

Re: vm with usb passtrought won't work after kernel upgrade

#11 Post by daggs »

CwF wrote:By 'sometimes' I don't mean sporadic on one machine, but works here in one machines configuration and does not work in another with differing kernel and in the case of the unifying receiver, the package solaar I think it was.
my issue happens on boot, not after sometime
You mean when you boot the vm, yes? So the usb is pre-defined to pass, the check if it is available at boot causes the vm to fail. What if you pass it after the vm is up and running without it? I believe this would test if can release from the host. I have two things that worked prior to 4.4x. IIRC I had a tv dongle that would pass under 3.x that now requires the port be passed. Don't know? At that time it was pci-stub and that maybe didn't work until vfio was in 3.9 or something?

I suppose you could try to blacklist or unload anything on the host. Look for something new that has change, perhaps before the host had no driver for the device and now it does?

I ask on the eth since I'm still looking for a 4 port card that can divy up 4 ports, mine does two pairs... I also use a dual port that can divide them, point being you need another slot for a usb card!?
has none. all the pci slots are taken.
I think I'll take this question to the libvirt ml, see what they have to say.

thanks for the help.

CwF
Global Moderator
Global Moderator
Posts: 2638
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 192 times

Re: vm with usb passtrought won't work after kernel upgrade

#12 Post by CwF »

daggs wrote: has none. all the pci slots are taken.
I think I'll take this question to the libvirt ml, see what they have to say.
Yes, I see, hence the suggestion of combining 4 cards into 2 or 1, leaving room for a usb card. Future note; not all usb cards vfio pass well, its own subject.

Please post back if there is another solution. Good luck.

daggs
Posts: 17
Joined: 2018-04-21 16:12

Re: vm with usb passtrought won't work after kernel upgrade

#13 Post by daggs »

Solved!

craparmor was to blame, I've disabled it in kernel params and all works again.

Post Reply