Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

Permission denied starting VM using block device on KVM

Linux Kernel, Network, and Services configuration.
Post Reply
Message
Author
programmersdigest
Posts: 8
Joined: 2018-09-09 10:42

Permission denied starting VM using block device on KVM

#1 Post by programmersdigest »

Hi @all,

I am running KVM on Debian 9.5 and want to use a block device in one of my VMs. Trying to start the VM (which is called "nas") I always get a permission denied error:

Code: Select all

virsh start nas
error: Failed to start domain nas
error: internal error: process exited while connecting to monitor: 2018-09-09T11:52:29.761625Z qemu-system-x86_64: -drive file=/dev/disk/by-id/ata-HGST_HTS541010A9E680_JD100ACC2MG2VK-part1,format=raw,if=none,id=drive-virtio-disk1,cache=none: Could not open '/dev/disk/by-id/ata-HGST_HTS541010A9E680_JD100ACC2MG2VK-part1': Permission denied
The block device configuration looks as follows:

Code: Select all

<disk type='block' device='disk'>
  <driver name='qemu' type='raw' cache='none'/>
  <source dev='/dev/disk/by-id/ata-HGST_HTS541010A9E680_JD100ACC2MG2VK-part1'/>
  <target dev='vdb1' bus='virtio'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</disk>
I am starting the VM as non-root user (however starting it as root doesn't seem to work, either.

Can anyone tell me what permissions are missing and how to resolve this? I'm at a loss...

p.H
Global Moderator
Global Moderator
Posts: 3049
Joined: 2017-09-17 07:12
Has thanked: 5 times
Been thanked: 132 times

Re: Permission denied starting VM using block device on KVM

#2 Post by p.H »

A standard user does not have permission to open a raw block device. The user must be a member of the group "disk", but this is of course dangerous and I would not recommend it.

Is the error message the same when starting the VM as root ?

programmersdigest
Posts: 8
Joined: 2018-09-09 10:42

Re: Permission denied starting VM using block device on KVM

#3 Post by programmersdigest »

Well, I cannot start my VM as root, because

Code: Select all

virsh list --all
does not show any entries when I am logged in as root. This is strange, isn't it?

I tried running qemu as root by uncommenting

Code: Select all

user = "root"
group = "root"
in the qemu.conf file. That didn't help.

I also tried adding the drive and partition to the cgroup_device_acl list (even though the comment says all block disk devices need not be added):

Code: Select all

cgroup_device_acl = [
    "/dev/sdb",
    "/dev/sdb1",
    "/dev/disk/by-id/ata-HGST_HTS541010A9E680_JD100ACC2MG2VK",
    "/dev/disk/by-id/ata-HGST_HTS541010A9E680_JD100ACC2MG2VK-part1"
]
Didn't help either.

Edit: I just found out that my domain configuration files were all stored in my users home directory. I guess this is because libvirt is not running as root user. I'll try to create a VM as root and access the block device from there.

User avatar
GarryRicketson
Posts: 5644
Joined: 2015-01-20 22:16
Location: Durango, Mexico

Re: Permission denied starting VM using block device on KVM

#4 Post by GarryRicketson »

Well, I am doing some searches, and am finding various things it could be, since I don't really know the answer here, but any way there is a couple of things I found, 1st, I would like to point out :
I tried running qemu as root by uncommenting
CODE: SELECT ALL
user = "root"
group = "root"
You say it didn't work any way, but you may want comment those lines again.
From: https://github.com/jedi4ever/veewee/issues/996
I don't think it is wise to make such edits letting qemu running as root. You should put the ISO file in a directory that is readable by qemu user, for example /tmp
I agree, here. There also is this comment:
If you copy your ISO image to /var/lib/libvirt/images (and access it from there) all the problems will go away.
Also, this : https://wiki.archlinux.org/index.php/KVM
Goes into some steps you can apply for trouble shooting.
====================
virsh list --all

does not show any entries when I am logged in as root.
Does that command show any entries if you run it as a normal user ?

For the searches, I copy pasted :

Code: Select all

Permission denied starting VM using block device on KVM 
into my search bot, (search engine), ... I don't have time to go through all the results just now.

Side note,: I do not ever need to be root to start any of my VM's, using QEMU,
That does not seem like a good idea to me.
Last edited by GarryRicketson on 2018-09-09 13:16, edited 1 time in total.

programmersdigest
Posts: 8
Joined: 2018-09-09 10:42

Re: Permission denied starting VM using block device on KVM

#5 Post by programmersdigest »

I got it to work now. I now let qemu run as user "root" and group "root". I also created and started the VM as root. Now the device can be accessed and used within the VM.

However, I'm not happy with this. I would rather have the VMs running as my normal user - they don't need root access for anything else.

My goal is to run an SMB share on this VM (hence the name "nas") and use a block device for maximum disk I/O.

So, is there a better way to allow a VM to use a block device? Maybe I could allow access to the libvirt group using udev rules? Or else, is there a better way to have decently fast filesystem access without using a block device?

@GarryRicketson: I agree, running as root is not what I want. However, I do not use an ISO image in this case - accessing my ISOs and qcow2 drives works fine - I want to pass a device directly through to the VM. And that seems to work only as root user. Yeah, I searched through those search results for a day now, without success. But I'm not a debian guru by any means, so I'm sure there is a way I just don't see...

Edit:
The user must be a member of the group "disk"
I only now realized what you were saying there, and you're right: Adding my user to the group "disk" does work. That's better than running as root I suppose. But still not perfect. Now if I could get those udev rules to work...

CwF
Global Moderator
Global Moderator
Posts: 2638
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 192 times

Re: Permission denied starting VM using block device on KVM

#6 Post by CwF »

You can simply link the disk to libvirt storage.

Using a root terminal make a link of /dev/disk/by-id/* in /var/lib/libvirt/images.

Leave groups and permissions alone for the most part. Reference storage only from libvirt/images. Create links as root in /images to anywhere you'd like.

Not only blocks by-id as you've figured out, but also any ISO's can be linked in and backing files also work well to reside elsewhere, but linked in.

programmersdigest
Posts: 8
Joined: 2018-09-09 10:42

Re: Permission denied starting VM using block device on KVM

#7 Post by programmersdigest »

@CwF but the link would have the same permissions as the source, would it not?

CwF
Global Moderator
Global Moderator
Posts: 2638
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 192 times

Re: Permission denied starting VM using block device on KVM

#8 Post by CwF »

I thought I saw it worked? At default qemu, kvm and libvirt have permission to anything in libvirt/images. The link provides permissions on behalf. So libvirt links on behalf of root to wherever.

programmersdigest
Posts: 8
Joined: 2018-09-09 10:42

Re: Permission denied starting VM using block device on KVM

#9 Post by programmersdigest »

@CwF: Yeah, I initially thought it worked, but it didn't after a reboot. I had set the permissions on the link, which in turn set them on /dev/sdb1. This did however not persist after reboot. AFAIK Debian always ignores symlink permissions, does it not?

Anyways, I got it to work using some udev-magic:
I added the following rule to a new file "/etc/udev/rules.d/10-libvirt.rules":

Code: Select all

KERNEL=="sd?1", SUBSYSTEMS=="scsi", ATTRS{model}=="HGST HTS541010A9", GROUP:="libvirt", MODE="0660"
This sets the group for the specific partition I want to use as a block device. Since /dev/disk/by-id/... is just a link to the specific device, the permissions carry over and I can use it in my VM config.

I'm happy with this, sure was a hard nut to crack. Thanks for the help @all :D

CwF
Global Moderator
Global Moderator
Posts: 2638
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 192 times

Re: Permission denied starting VM using block device on KVM

#10 Post by CwF »

programmersdigest wrote:AFAIK Debian always ignores symlink permissions, does it not?
Not sure really. It's the way it works for me! Glad you got it working to satisfaction. The udev trick looks good. Awhile back I found pools of storage outside libvit/images even created with VMManager caused permission issues with virsh and guestmount and the like where the user has permission for the command but not the path listed within. So don't know why this solution did not persist for you. Half the images in my directory are such links.

p.H
Global Moderator
Global Moderator
Posts: 3049
Joined: 2017-09-17 07:12
Has thanked: 5 times
Been thanked: 132 times

Re: Permission denied starting VM using block device on KVM

#11 Post by p.H »

programmersdigest wrote:Adding my user to the group "disk" does work. That's better than running as root I suppose.
IMO it is worse than running the VM as root. Now your user has the power to read and destroy the contents of any disk or partition.
programmersdigest wrote:AFAIK Debian always ignores symlink permissions, does it not?
It's not Debian but the Linux kernel. Symlink permissions are irrelevant, only the permissions on the final target matter.

CwF
Global Moderator
Global Moderator
Posts: 2638
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 192 times

Re: Permission denied starting VM using block device on KVM

#12 Post by CwF »

It seems to me there must be other goofery going on. I likely would not try any direct changes to groups and permissions to get it to work. For the permissions "on behalf" I don't really imply the permission flows through, ie, the user has the rights to ask libvirt, and libvirt has the rights. My user can not directly access the drive for example. My only example coming to mind, with guestmount the user can not call out a direct path, however with the -d domain switch the user can call on the disk. So, the switch allows "on behalf".
Why a link would not persist across a reboot makes no sense. And when you change or add another such thing will a trick again be needed? I simply open a root thunar and right drag the by-id file to libvirt/image and select create link. I don't need to modify the permissions...
Something else is different here. I'm hoping my system is the one set up right...?

I think I'd like to see his user groups, for the one operating as dom0.

programmersdigest
Posts: 8
Joined: 2018-09-09 10:42

Re: Permission denied starting VM using block device on KVM

#13 Post by programmersdigest »

I could only get it to work having the VM started by root. It did not suffice to have libvirt run as root. Which I think is the point CwF is also making: my user shouldn't need to have access rights to mount the drive. However, on my system, for some reason it doesn't seem to matter much which account libvirt is running as. It seems to be the user executing virt-install whos access rights matter. I don't think this is the way it should be... but I didn't really change anything. It's pretty much a fresh and standard Debian installation.

I don't have the system at hand right now, but the user in question should only have the groups "libvirt", "libvirt-qemu" and "kvm".

Btw. I may have been unclear in my statement before: the symlink I create DOES persist. However changing access rights using chown or chmod on /dev/sdb1 does not. From what I can gather these get overwritten at reboot.

CwF
Global Moderator
Global Moderator
Posts: 2638
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 192 times

Re: Permission denied starting VM using block device on KVM

#14 Post by CwF »

programmersdigest wrote:the symlink I create DOES persist. However changing access rights using chown or chmod on /dev/sdb1 does not. .
...this changing rights I don't do. If I were to add a disk, maybe in my dock, I'd open a root thunar with two tabs, one /dev/disk/by-id/ and /var/lib/libvirt/images. The new disk file is then right-click dragged to images and 'create link'. Then with VMManager, refresh the storage pool and pick the disk. Done.

Here for example the user can not use 'virsh attach-disk' and call a file in /images = path permission denied. VMM is the 'on behalf' operator there.

Similar, in using a link, I have a menu to add 'disk' to vm's using usb for temporary. In this case, the link is done the same way, the user does have access to the location storing the images, and the image. However the 'virsh attach-disk' used in the menu requires sudo (gksu) to allow access to the link in /images. My point here is user rights to the source is irrelevant.

p.H
Global Moderator
Global Moderator
Posts: 3049
Joined: 2017-09-17 07:12
Has thanked: 5 times
Been thanked: 132 times

Re: Permission denied starting VM using block device on KVM

#15 Post by p.H »

programmersdigest wrote:However changing access rights using chown or chmod on /dev/sdb1 does not. From what I can gather these get overwritten at reboot.
/dev is a special temporary filesystem (devtmpfs). Devices files are dynamically created with default permissions each time the corresponding device is "added" by the kernel, and deleted when the device is removed. I guess you can change the default permissions with udev rules.

programmersdigest
Posts: 8
Joined: 2018-09-09 10:42

Re: Permission denied starting VM using block device on KVM

#16 Post by programmersdigest »

I guess you can change the default permissions with udev rules
Yup, that works. But everything I can find regarding the use of block devices with kvm seems to indicate that there is no need to change any permissions. So I'm a little confused as to what it is I'm doing wrong.

programmersdigest
Posts: 8
Joined: 2018-09-09 10:42

Re: Permission denied starting VM using block device on KVM

#17 Post by programmersdigest »

Oooh, I think I understand it now: most people control their VMs using virt-manager, which uses qemu://system to connect. That in turn connects to the libvirt daemon, which is usually run as root. I use a non privileged user. In this case virsh connects to qemu://session and a new instance of libvirt is started for that user. That's why my users needs special permissions. I will have to think about how to do it right now.

CwF
Global Moderator
Global Moderator
Posts: 2638
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 192 times

Re: Permission denied starting VM using block device on KVM

#18 Post by CwF »

I appreciate the eyes on this. It would be good to have this ironed out. It bugs me to have unknowns in my own understanding here.

I do remember permission issues setting mine up but think it was just the groups for my dom0.

Code: Select all

$ groups
dom0 adm cdrom floppy sudo audio dip video plugdev netdev kvm libvirt libvirt-qemu
Next, as noted I've seen that permissions are different using cli than using a front end like VMM. For me it's simpler to use VMM only. Originally I would have the storage path be the direct path to /dev and VMM made a new pool to the root file system automatically. It worked fine but that bugged me. During experimentation with the storage pools I decided to have only the default pool and used links. Never saw a how-to on that, just tried it and it worked fine, and it simplified the VMM's storage pools.

That was a few years ago on a 8.1 machine. My current main machine started as 8.8 and is upgraded to ~9.5. No change in behavior. I'm pretty sure setting that one up I added the above groups to the user and that's it.

So clarify if your using VMM, or other, or virsh?

ADD: and I just saw your response.....I'll send anyway.

Post Reply