Using network namespaces with veth to NAT guests with overlapping IPs

Sets of virtual machines are connected to a virtual bridges (e.g. virbr0 and virbr1) and as they are isolated, can use the same subnet range and set of IPs. However, NATing becomes a problem because the host won’t know which VM to return the traffic to.

To solve this problem, we can use network namespaces and some veth (virtual Ethernet) devices to connect up each private network we want to NAT.

Each veth device acts like a patch cable and is actually made up of two network devices, one for each end (e.g. peer1-a and peer1-b). By adding those interfaces between bridges and/or namespaces, you create a link between them.

The network namespace is only used for NAT and is where the veth IPs are set, the other end will act like a patch cable without an IP. The VMs are only connected into their respective bridge (e.g. virbr0) and can talk to the network namespace over the veth patch.

We will use two pairs for each network namespace.

• One (e.g. represented by veth1 below ) which connects the virtual machine’s private network (e.g. virbr0 on 10.0.0.0/24) into the network namespace (e.g. net-ns1) where it sets an IP and will be the private network router (e.g. 10.0.0.1).
• Another (e.g. represented by veth2 below) which connects the upstream provider network (e.g. br0 on 192.168.0.0/24) into the same network namespace where it sets an IP (e.g. 192.168.0.100).
• Repeat the process for other namespaces (e.g. represented by veth3 and veth4 below).

By providing each private network with is own unique upstream routable IP and applying NAT rules inside each namespace separately we can avoid any conflict.

Create a provider bridge

You’ll need a bridge to a physical network, which will act as your upstream route (like a “provider” network).

ip link add name br0 type bridge
ip link set br0 up
ip link set eth0 up
ip link set eth0 master br0

Create namespace

We create our namespace to patch in the veth devices and hold the router and isolated NAT rules. As this is for the purpose of NATing multiple private networks, I’m making it sequential and calling this nat1 (for our first one, then I’ll call the next one nat2).

ip netns add nat1

First veth pair

Our first veth peer interfaces pair will be used to connect the namespace to the upstream bridge (br0). Give them a name that makes sense to you; here I’m making it sequential again and specifying the purpose. Thus, peer1-br0 will connect to the upstream br0 and peer1-gw1 will be our routable IP in the namespae.

ip link add peer1-br0 type veth peer name peer1-gw1

Adding the veth to provider bridge

Now we need to add the peer1-br0 interface to the upstream provider bridge and bring it up. Note that we do not set an IP on this, it’s a patch lead. The IP will be on the other end in the namespace.

brctl addif br0 peer1-br0
ip link set peer1-br0 up

First gateway interface in namespace

Next we want to add the peer1-gw1 device to the namespace, give it an IP on the routable network, set the default gateway and bring the device up. Note that if you use DHCP you can do that, here I’m just setting an IP statically to 192.168.0.100 and gateway of 192.168.0.1.

ip link set peer1-gw1 netns nat1
ip netns exec nat1 ip addr add 192.168.0.100/24 dev peer1-gw1
ip netns exec nat1 ip link set peer1-gw1 up
ip netns exec nat1 ip route add default via 192.168.0.1

Second veth pair

Now we create the second veth pair to connect the namespace into the private network. For this example we’ll be connecting to virbr0 network, where our first set of VMs are running. Again, give them useful names.

ip link add peer1-virbr0 type veth peer name peer1-gw2

Adding the veth to private bridge

Now we need to add the peer1-virbr0 interface to the virbr0 private network bridge. Note that we do not set an IP on this, it’s a patch lead. The IP will be on the other end in the namespace.

brctl addif virbr0 peer1-virbr0
ip link set peer1-virbr0 up

Second gateway interface in namespace

Next we want to add the peer1-gw2 device to the namespace, give it an IP on the private network and bring the device up. I’m going to set this to the default gateway of the VMs in the private network, which is 10.0.0.1.

ip link set peer1-gw2 netns nat1
ip netns exec nat1 ip addr add 10.0.0.1/24 dev peer1-gw2
ip netns exec nat1 ip link set up dev peer1-gw2

Enable NAT in the namespae

So now we have our namespace with patches into each bridge and IPs on each network. The final step is to enable network address translation.

ip netns exec nat1 iptables -t nat -A POSTROUTING -o peer1-gw1 -j MASQUERADE
ip netns exec nat1 iptables -A FORWARD -i peer1-gw1 -o peer1-gw2 -m state --state RELATED,ESTABLISHED -j ACCEPT
ip netns exec nat1 iptables -A FORWARD -i peer1-gw2 -o peer1-gw1 -j ACCEPT

You can see the rules with standard iptables in the netspace.

ip netns exec nat1 iptables -t nat -L -n

Test it

OK so logging onto the VMs, they should a local IP (e.g. 10.0.0.100, a default route to 10.0.0.1 and have upstream DNS set. Test that they can ping the gateway, test they can ping the DNS and test that they can ping a DNS name on the Internet.

Rinse and repeat

This can be applied for other virtual machine networks as required. There is no-longer any need for the VMs there to have unique IPs, they can overlap eachother.

What you do need to do is create a new network namespace, create two new sets of veth pairs (with a useful name) and pick another IP on the routable network. The virtual machine gateway IP will be the same in each namespace, that is 10.0.0.1.