KVM + VLAN-aware-bridge

[aquilares]

Hi all,



I wanted to ask this here, since I've been banging my head on this for a while and I can't seem to find 2 how-to's online which are exactly the same.

So, my situation:

• I have a system with 2 interfaces.

• One interface will be a normal interface with a normal bridge, nothing fancy, let's call this the WAN-interface.

• The second interface is the internal LAN-interface, this will be the topic of this thread: should be a VLAN-aware bridge.

• This system is going to be running a virtual router/firewall, in my case OPNsense. It should have multiple VLAN's on that internal LAN interface.

I was thinking of doing the following on my debian KVM hypervisor (I left out the part of the WAN-bridge, that one is easy to do):

Code: Select all

auto lo
iface lo inet loopback

auto enp1s0
iface enp1s0 inet manual

auto brlan
iface brlan inet manual
	bridge-ports enp1s0 
	bridge-stp off
	bridge-fd 0
	bridge-vlan-aware yes
	bridge-vids 2-4094

auto brlan.1
iface brlan.1 inet static
	address 192.168.1.126/24
	gateway 192.168.1.1
	
source /etc/network/interfaces.d/*

The goal is:

• I want to pass multiple VLAN's to that LAN-bridge which is VLAN-aware.

• The hypervisor itself also has an interface in one of the VLAN's, for remote management (brlan.1)

• This physical interface is connected to a trunk port on a managed switch which is passing all the necessary VLAN's on as tagged VLAN's.

My question now is: what more do I need to do to get this working?
I know I'm going to have to configure the VLAN's themselves INSIDE of the opnsense VM, not on the hypervisor level. That much I understand.

Is there anything more I need to do on the hypervisor level to get these VLAN's inside of the VM?

• I see some tutorials talking about messing with net.ipv4.conf.all.arp_filter=0 etc., is this necessary for this kind of setup?

• I see a lot of tutorials involving proxmox, where they "need to tag the VM interface also on the level of the hypervisor it seems". Is that necessary here and how to do this then?

• I see some tutorials doing an extra step like "bridge vlan add dev enp1s0 vid 10". Is this also necessary to get the VLAN's inside of the VM?

[fabien]

Hello,

does this help? [Solved] Complex bond/bridge setup

[aquilares]

Thanks, I already found and read that thread.

However, if you look at the last example of the config in that thread, for me it's still not clear what he's doing there.

He's configuring a vlan on top of his bridge in 2 different ways: huh?

You have his bridge20 interface on top of vlan20 on top of br0 and his br0.1 on top of br0, both vlans on top of bridges?
And then what? Does he assign his VM-interface to br0 or to bridge20?

The statement he is making there: "KVM/QEMU does not accept the bridge notation with dots in it, such as bridge0.20" => the first and only time I ever encounter this statement somewhere, is this even correct?

[lindi]

I see many options:

• Create VLAN interfaces using ifupdown. For each VLAN, also create a bridge. Then add each VM to the appropriate bridge. If you really want to support native VLAN, you probably need to put all of this on top of a bridge but is that really a good idea? I would keep trunk port only for tagged packets. Note that in this case it is easy to allow VMs to connect to your hypervisor's management IP even if you did not intend this due to Linux's weak host model.

• Use openvswitch, no experience on that

• Use SR-IOV. This requires hardware support but basically allows you to split your physical network hardware into virtual network devices that you can then directly passthrough to each VM bypassing the hypervisor network stack entirely for high performance.

[aquilares]

Hi lindi, thanks for the feedback.

I'm aware of the other options, like openvswitch or the classical linux-bridge-per-vlan interface.
However, I would like to avoid using openvswitch, I rather rely on native kernel components, not another extra piece of software which could go wrong.
Also: with classical bridges, every time you need to add a new vlan, I would need to:

• add vlan on switch trunk

• add vlan interface on hypervisor

• add bridge on vlan interface on hypervisor

• add new bridge to vm

• reboot vm

• configure interface in vm

With vlan aware bridge:

• add vlan on trunk on switch

• add vlan inside VM

• configure vlan interface in VM

• done

So again, I'm looking for specific how-to on how to use vlan-aware bridge together with KVM hypervisor.

SR-IOV isn't an option here, not on my hardware.

[lindi]

As far as I know, libvirt does not support this. I would be happy to be proven wrong. Or do you mean you would add the VLAN tag inside the VM? This sounds like a nightmare to maintain as the VM could then easily spoof the VLAN tag?

[aquilares]

Yep that's what I meant.
Ah seriously, libvirt doesn't support this?

[lindi]

You can add the tag inside the VM but then that is up to each guest OS to implement and it won't provide you security benefits unless you trust the guest OS.

[aquilares]

You can add the tag inside the VM but then that is up to each guest OS to implement and it won't provide you security benefits unless you trust the guest OS.

Now, if you consider the other option, using the classic bridge-per-vlan interface setup:

• isn't it a pain to manage all those bridges on the level of the hypervisor?

• Here I need to trust the hypervisor? If someone gains access to my debian-based hypervisor and changes the assignment of the vlans to bridges...

• Isn't there a limit on the number of bridges/vnic's per VM in Libvirt/KVM? Based on my initial setup, I'm probably already going to be dealing with at least 6 or 7 bridges...

[lindi]

• isn't it a pain to manage all those bridges on the level of the hypervisor?

• Here I need to trust the hypervisor? If someone gains access to my debian-based hypervisor and changes the assignment of the vlans to bridges...

• Isn't there a limit on the number of bridges/vnic's per VM in Libvirt/KVM? Based on my initial setup, I'm probably already going to be dealing with at least 6 or 7 bridges...

Yes it's a pain to manage. But probably even worse if you have to manage it inside each guest OS. What if you need to renumber some VLAN? How do you configure VMs that are not currently running? Also, you already need to trust the hypervisor, it can do anything. I don't think there is any limit on how many interfaces a VM can have.

[aquilares]

• isn't it a pain to manage all those bridges on the level of the hypervisor?

• Here I need to trust the hypervisor? If someone gains access to my debian-based hypervisor and changes the assignment of the vlans to bridges...

• Isn't there a limit on the number of bridges/vnic's per VM in Libvirt/KVM? Based on my initial setup, I'm probably already going to be dealing with at least 6 or 7 bridges...

Yes it's a pain to manage. But probably even worse if you have to manage it inside each guest OS. What if you need to renumber some VLAN? How do you configure VMs that are not currently running? Also, you already need to trust the hypervisor, it can do anything. I don't think there is any limit on how many interfaces a VM can have.

Interesting, I'll reconsider this option.

But also here, any idea if there is any need to "fiddle" with other settings to get this working?

Getting back to my initial post, I guess the config would be something like this:

Code: Select all

auto lo
iface lo inet loopback

auto enp1s0
iface enp1s0 inet manual

auto enp1s0.10
iface enp1s0.10 inet manual

auto enp1s0.20
iface enp1s0.20 inet manual

auto enp1s0.30
iface enp1s0.30 inet manual

auto vlan10
iface vlan10 inet static
	address 192.168.10.126/24
	gateway 192.168.10.1
	bridge-ports enp1s0.10
	bridge-stp off
	bridge-fd 0
	
auto vlan20
iface vlan20 inet manual
	bridge-ports enp1s0.20
	bridge-stp off
	bridge-fd 0
	
auto vlan30
iface vlan30 inet manual
	bridge-ports enp1s0.30
	bridge-stp off
	bridge-fd 0
	
source /etc/network/interfaces.d/*

Does this look correct?
So multiple vlan interfaces without IP, then for every vlan interface a bridge defined.
Those bridges all do NOT have an IP, since I don't want my hypervisor to have an IP in each vlan, EXCEPT for vlan10, where I configure an IP on the bridge to be able to SSH into the hypervisor for management.

I then configure the VM with multiple vnic's, one linked to each of the bridges.
Result: no VLAN config in the VM, for the VM they seem normal non-vlan interfaces.

Is there anything else then which needs to happen?
E.g. configuring settings such as "net.ipv4.conf.all.arp_filter=0"?

When I read this, it seems to be necessary?
https://computingpost.medium.com/create ... 679e3894bd

[lindi]

Those bridges all do NOT have an IP, since I don't want my hypervisor to have an IP in each vlan, EXCEPT for vlan10, where I configure an IP on the bridge to be able to SSH into the hypervisor for management.

This is what I was trying to warn you about. IP addresses are not strongly tied to interface in Linux. You can connect to the IP even over an interface that has no IP address assigned.

[aquilares]

Those bridges all do NOT have an IP, since I don't want my hypervisor to have an IP in each vlan, EXCEPT for vlan10, where I configure an IP on the bridge to be able to SSH into the hypervisor for management.

This is what I was trying to warn you about. IP addresses are not strongly tied to interface in Linux. You can connect to the IP even over an interface that has no IP address assigned.

OK, first time I hear about this problem?
Would this be different using openvswitch?

[lindi]

OK, first time I hear about this problem?
Would this be different using openvswitch?

You can check https://en.wikipedia.org/wiki/Host_model -- I belive openvswitch should make this easier.

[aquilares]

OK, first time I hear about this problem?
Would this be different using openvswitch?

You can check https://en.wikipedia.org/wiki/Host_model -- I belive openvswitch should make this easier.

Woops, guess not.
Just found this:
https://docs.openvswitch.org/en/stable/faq/vlan/
Look for "I configured one IP address on VLAN 0 and another on VLAN 9, like this:"