Tag Archives: bridge

How to create VLAN trunks and access ports for VMs on Linux bridges using NetworkManager (and have them talk)

TL;DR instead of creating a VLAN on a physical interface (like bond0.123), then turning that into a bridge for a VM, create a tagged interface on the main bridge (e.g. br0.123) and then add that into a new bridge for a VM.

Create a new bridge for the VLAN.

nmcli con add ifname br-123 type bridge con-name br-123
nmcli con modify br-123 ipv4.method disabled ipv6.method ignore
nmcli con up br-123

Create the VLAN on existing bridge (assuming br0 already exists) and attach to the new bridge (br-123).

nmcli con add type vlan con-name br0.123 ifname br0.123 dev br0 id 123
nmcli con modify br0.123 master br-123 slave-type bridge
nmcli con up br0.123

This will allow two VMs on the same host to talk to each other over the VLAN, where one is using a tagged interface on br0 (as a trunk) and the other is using br-123 (as an access port or native for VLAN 123).

+------+  +-------+                  +------------------+
| eth0 |--|       |                  |       VM1        |
+------+  |       |   +----------+   |   +----------+   |
          | bond0 |---|   br0    |---|---| eth0.123 |   |
+------+  |       |   | (bridge) |   |   |  (VLAN)  |   |
| eth1 |--|       |   +----------+   |   +----------+   |
+------+  +-------+        |         +------------------+
                           |
                           |                       +------------------+
                           |                       |       VM2        |
                      +---------+   +----------+   |   +----------+   |
                      | br0.123 |---|  br-123  |---|---|   eth0   |   |
                      | (VLAN)  |   | (bridge) |   |   | (native) |   |
                      +---------+   +----------+   |   +----------+   |
                                                   +------------------+

Some background

A common method for connecting VMs to a real network is to attach them to a bridge. In this case, a bridge is created on the KVM host and a physical interface is attached to it. Network interfaces of the VMs are then attached to that bridge and they are directly on the same network as the host (we’ll call this a flat network).

+------+  +-------+                  +----------------+
| eth0 |--|       |                  |       VM       |
+------+  |       |   +----------+   |   +--------+   |
          | bond0 |---|   br0    |---|---|  eth0  |   |
+------+  |       |   | (bridge) |   |   | (flat) |   |
| eth1 |--|       |   +----------+   |   +--------+   |
+------+  +-------+                  +----------------+

That’s great, but now they’re also connected to everything else on the network and maybe you didn’t want that. One solution to this is to use VLANs to put VMs on isolated networks.

The bridge mentioned above could also be used as a trunk, which can carry tagged VLAN traffic that the VM creates (assuming the physical switch is configured to accept those VLANs). In this instance, the VM would create a VLAN on its network interface and tagged traffic will flow out of the bridge onto the physical network, letting that VM talk to other VMs or devices on that same VLAN.

Continue reading

How to create bridges on bonds (with and without VLANs) using NetworkManager

Some production systems you face might make use of bonded network connections that you need to bridge in order to get VMs onto them. That bond may or may not have a native VLAN (in which case you bridge the bond), or it might have VLANs on top (in which case you want to bridge the VLANs), or perhaps you need to do both.

Let’s walk through an example where we have a bond that has a native VLAN, that also has the tagged VLAN 123 on top (and maybe a second VLAN 456), all of which need to be separately bridged. This means we will have the bond (bond0) with a matching bridge (br-bond0), plus a VLAN on the bond (bond0.123) with its matching bridge (br-vlan123). It should look something like this.

+------+   +---------+                           +---------------+
| eth0 |---|         |          +------------+   |  Network one  |
+------+   |         |----------|  br-bond0  |---| (native VLAN) |
           |  bond0  |          +------------+   +---------------+
+------+   |         |                                            
| eth1 |---|         |                                            
+------+   +---------+                           +---------------+
            | |   +---------+   +------------+   |  Network two  |
            | +---| vlan123 |---| br-vlan123 |---| (tagged VLAN) |
            |     +---------+   +------------+   +---------------+
            |                                                     
            |     +---------+   +------------+   +---------------+
            +-----| vlan456 |---| br-vlan456 |---| Network three |
                  +---------+   +------------+   | (tagged VLAN) |
                                                 +---------------+

To make it more complicated, let’s say that the native VLAN on the bond needs a static IP and to operate at an MTU of 1500 while the other uses DHCP and needs MTU of 9000.

OK, so how do we do that?

Continue reading

How to create Linux bridges and Open vSwitch bridges with NetworkManager

My virtual infrastructure Ansible role supports connecting VMs to both Linux and Open vSwitch bridges, but they must already exist on the KVM host.

Here is how to convert an existing Ethernet device into a bridge. Be careful if doing this on a remote machine with only one connection! Make sure you have some other way to log in (e.g. console), or maybe add additional interfaces instead.

Export interfaces and existing connections

First, export the the device you want to convert so we can easily reference it later (e.g. eth1).

export NET_DEV="eth1"

Now list the current NetworkManager connections for your device exported above, so we know what to disable later.

sudo nmcli con |egrep -w "${NET_DEV}"

This might be something like System eth1 or Wired connection 1, let’s export it too for later reference.

export NM_NAME="Wired connection 1"

Create a Linux bridge

Here is an example of creating a persistent Linux bridge with NetworkManager. It will take a device such as eth1 (substitute as appropriate) and convert it into a bridge. Note that we will be specifically giving it the device name of br0 as that’s the standard convention and what things like libvirt will look for.

Continue reading

Configuring QEMU bridge helper after “access denied by acl file” error

QEMU has a neat bridge-helper utility which allows a non-root user to easily connect a virtual machine to a bridged interface. In Fedora at least, qemu-bridge-helper runs as setuid (any user can run as root) and privileges are immediately dropped to cap_net_admin. It also has a simple white/blacklist ACL mechanism in place which limits connections to virbr0, libvirt’s local area network.

That’s all great, but often you actually want a guest to be a part of your real network. This means it must connect to a bridged interface (often br0) on a physical network device.

Continue reading