TL;DR instead of creating a VLAN on a physical interface (like bond0.123), then turning that into a bridge for a VM, create a tagged interface on the main bridge (e.g. br0.123) and then add that into a new bridge for a VM.
Create a new bridge for the VLAN.
nmcli con add ifname br-123 type bridge con-name br-123
nmcli con modify br-123 ipv4.method disabled ipv6.method ignore
nmcli con up br-123
Create the VLAN on existing bridge (assuming br0 already exists) and attach to the new bridge (br-123).
nmcli con add type vlan con-name br0.123 ifname br0.123 dev br0 id 123
nmcli con modify br0.123 master br-123 slave-type bridge
nmcli con up br0.123
This will allow two VMs on the same host to talk to each other over the VLAN, where one is using a tagged interface on br0 (as a trunk) and the other is using br-123 (as an access port or native for VLAN 123).
+------+ +-------+ +------------------+
| eth0 |--| | | VM1 |
+------+ | | +----------+ | +----------+ |
| bond0 |---| br0 |---|---| eth0.123 | |
+------+ | | | (bridge) | | | (VLAN) | |
| eth1 |--| | +----------+ | +----------+ |
+------+ +-------+ | +------------------+
|
| +------------------+
| | VM2 |
+---------+ +----------+ | +----------+ |
| br0.123 |---| br-123 |---|---| eth0 | |
| (VLAN) | | (bridge) | | | (native) | |
+---------+ +----------+ | +----------+ |
+------------------+
Some background
A common method for connecting VMs to a real network is to attach them to a bridge. In this case, a bridge is created on the KVM host and a physical interface is attached to it. Network interfaces of the VMs are then attached to that bridge and they are directly on the same network as the host (we’ll call this a flat network).
+------+ +-------+ +----------------+
| eth0 |--| | | VM |
+------+ | | +----------+ | +--------+ |
| bond0 |---| br0 |---|---| eth0 | |
+------+ | | | (bridge) | | | (flat) | |
| eth1 |--| | +----------+ | +--------+ |
+------+ +-------+ +----------------+
That’s great, but now they’re also connected to everything else on the network and maybe you didn’t want that. One solution to this is to use VLANs to put VMs on isolated networks.
The bridge mentioned above could also be used as a trunk, which can carry tagged VLAN traffic that the VM creates (assuming the physical switch is configured to accept those VLANs). In this instance, the VM would create a VLAN on its network interface and tagged traffic will flow out of the bridge onto the physical network, letting that VM talk to other VMs or devices on that same VLAN.
+------+ +-------+ +--------------------+
| eth0 |--| | | VM |
+------+ | | +----------+ | +------------+ |
| bond0 |---| br0 |---|---| eth0.123 | |
+------+ | | | (bridge) | | | (VLAN) | |
| eth1 |--| | +----------+ | +------------+ |
+------+ +-------+ +--------------------+
This is handy because you might need a VM to talk to multiple networks without routing.
However, if all you wanted is for the VM to be on a single VLAN transparently without it requiring any configuration, then the above method is rather cumbersome as you have to set up VLAN interfaces. Instead, this can be achieved by creating a tagged VLAN on a physical network interface on the KVM host, then attaching that to a bridge. The VM would then be attached to the bridge as an access port and all traffic that flows out onto the network will be tagged (we’ll call this access port or native mode).
+------+ +-------+ +------------------+
| eth0 |--| | | VM |
+------+ | | +-----------+ +----------+ | +----------+ |
| bond0 |---| bond0.123 |---| br-123 |---|---| eth0 | |
+------+ | | | (VLAN) | | (bridge) | | | (native) | |
| eth1 |--| | +-----------+ +----------+ | +----------+ |
+------+ +-------+ +------------------+
Getting trunks talking to access ports
This is all well and good, but what if you had a VM on an access point wanting to talk to a VM on a trunk over the same VLAN? This won’t happen by default as the other VM would be on a bridge, not the bond. So, instead of creating the VLAN on the KVM host physical interface (e.g. bond0) as we did above, we need to create it on the same bridge we’re using for the trunk (e.g. br0), then add tagged interface that to a new bridge for the VM.
+------+ +-------+
| eth0 |--| |
+------+ | | +----------+
| bond0 |---| br0 |
+------+ | | | (bridge) |
| eth1 |--| | +----------+
+------+ +-------+ |
|
| +------------------+
| | VM |
+---------+ +----------+ | +----------+ |
| br0.123 |---| br-123 |---|---| eth0 | |
| (VLAN) | | (bridge) | | | (native) | |
+---------+ +----------+ | +----------+ |
+------------------+
Doing it this way allows us to also support both adding VMs directly to a VLAN or trunking in a number of VLANs and have the VMs able to communicate with each other.
+------+ +-------+ +------------------+
| eth0 |--| | | VM1 |
+------+ | | +----------+ | +----------+ |
| bond0 |---| br0 |---|---| eth0.123 | |
+------+ | | | (bridge) | | | (VLAN) | |
| eth1 |--| | +----------+ | +----------+ |
+------+ +-------+ | +------------------+
|
| +------------------+
| | VM2 |
+---------+ +----------+ | +----------+ |
| br0.123 |---| br-123 |---|---| eth0 | |
| (VLAN) | | (bridge) | | | (native) | |
+---------+ +----------+ | +----------+ |
+------------------+
Now VM1 and VM2 are both able to talk to the network on VLAN 123, as well as each other with VM1 via a tagged interface on the trunk and VM2 via an access port.
Multiple VLANs and access ports on the same VM
One of the downsides of the above approach is that the VM cannot also use the access port as a trunk, as all traffic will be re-tagged with a specific VLAN. If you want a VM to be on multiple networks with an access port, then you can mix and match by adding additional interfaces to it, some on the trunk with VLAN interfaces and others on access ports.
+-------------------+
+------+ +-------+ | VM |
| eth0 |--| | | +-----------+ |
+------+ | | +----------+ | | eth0.123 | |
| bond0 |---| br0 |---|---| eth0.456 | |
+------+ | | | (bridge) | | | (VLANs) | |
| eth1 |--| | +----------+ | +-----------+ |
+------+ +-------+ | | |
| | +----------+ |
| | | eth1 | |
| | | (native) | |
| | +----------+ |
| +-------|-----------+
| |
+---------+ +----------+
| br0.789 |---| br-789 |
| (VLAN) | | (bridge) |
+---------+ +----------+
OK, how do we do this?
Easy! I’m assuming that you already have a bridged interface (for example, br0) . If not, see this previous blog post.
- Create a new bridge on the KVM host to carry tagged packets for the access port (for example,
br-vlan123). - Create a new VLAN interface on our main bridge (for example,
br0.123) to be used for the access port. - Attach the VLAN interface to our new bridge.
- Bring the interfaces up.
- Attach our VM to the new bridge.
nmcli con add ifname br-vlan123 type bridge con-name br-vlan123
nmcli con modify br-vlan123 ipv4.method disabled ipv6.method ignore
nmcli con up br-vlan123
nmcli con add type vlan con-name br0.123 ifname br0.123 dev br0 id 123
nmcli con modify br0.123 master br-vlan123 slave-type bridge
nmcli con up br0.123
2 thoughts on “How to create VLAN trunks and access ports for VMs on Linux bridges using NetworkManager (and have them talk)”
Just wanted to let you know, this post was very helpful. It did bring up some additional questions through. One being, after creating the bridged vlan interface (br-vlan123), I do not see this available in Virt-Manager to attach as a network interface. Am I doing something wrong?
Also, why would one attach vlan’s to the bond (or single interface) vs a trunked bridge (br0)? Still trying to wrap my head as to the how’s and why’s with linux networking.
I think virt manager might be filtering on bridges which have `br[0-9]` but I’d have to check. If you put a VM directly on `br-vlan123` then that VM will be on a network and the host will tag the packets with VLAN 123 automatically when they leave the host, without the VM having to do anything. If you put a VM on a trunked bridge, then it needs to have a tagged VLAN interface. So I guess it’s about which way you prefer.