Need to configure second NIC to bridge LXC

Question

I installed Ubuntu 16.04 Server on a machine with 4 network cards. I have interfaces eth0 and eth1 connected to the same switch. The interface eth0 is meant for the remote SSH connection to manage the server. I want to use eth1 to be bridged by br0. This bridge I want to use for LXC containers. This setup in a DHCP environment did not cause me any problems. The challenge is that the network this server is installed in is fully static. I received an IP range for this server with same subnet mask and gateway.

Setting up eth0 was no problem:

auto eth0
iface eth0 inet static
   address   195.x.x.2
   network   195.x.x.0
   netmask   255.255.255.0
   gateway   195.x.x.1
   broadcast 195.x.x.255
   dns-nameservers 150.x.x.105 150.x.x.106

The problem comes with the second interface eth1, because it has the same gateway as eth0 Ubuntu warns that only one default gateway can be set (which is logical). Therefor I had set eth1 as follows:

auto eth1
iface eth1 net static
    address   195.x.x.3
    network   195.x.x.0
    netmask   255.255.255.0
    broadcast 195.x.x.255

Problem with this setup is that I can externally ping eth0 at IP 195.x.x.2 but eth1 cannot be pinged or accessed via SSH. I managed to make it work with a lot of routing trickery but as many articles write on this that this way is a hole which gets deeper if you have static bridge and containers for this.

My question is: Does anyone has a straight forward approach for my issue? How should I configure eth0 and eth1 to normally bridge the containers to eth1 with static IP numbers?

Answer 1

Ok I solved it in the following manner, by still proceeding with the gateway routing solution as described in the question. Maybe people with the same issue could use this approach as well or if somebody knows a better solution feel free to comment.

On the host:

I enabled ARP filtering:

sysctl -w net.ip4.conf.all.arp_filter=1
echo "net.ipv4.conf.all.arp_filter = 1" >> /etc/sysctl.conf

Configured the /etc/network/interfaces:

auto lo
iface lo net loopback

# The primary network interface
auto etc0
iface eth0 inet static
   address   195.x.x.2
   network   195.x.x.0
   netmask   255.255.255.0
   gateway   195.x.x.1
   broadcast 195.x.x.255
   up ip route add 195.x.x.0/24 dev eth0 src 195.x.x.2 table eth0table
   up ip route add default via 195.x.x.1 dev eth0 table eth0table
   up ip rule add from 195.x.x.2 table eth0table
   up ip route add 195.x.x.0/24 dev eth0 src 195.0.0.2
   dns-nameservers 150.x.x.105 150.x.x.106

# The secondary network interface
auto eth1
iface eth1 net manual

# LXC bridge interface
auto br0
iface br0 inet static
   address   195.x.x.3
   network   195.x.x.0
   netmask   255.255.255.0
   bridge_ifaces  eth1
   bridge_ports   eth1
   bridge_stp     off
   bridge_fd      0
   bridge_maxwait 0
   up ip route add 195.x.x.0/24 dev br0 src 195.x.x.3 table br0table
   up ip route add default via 195.x.x.1 dev br0 table br0table
   up ip rule add from 195.x.x.3 table br0table
   up ip route add 195.x.x.0/24 dev br0 src 195.0.0.3

Added the following lines to /etc/iproute2/rt_tables:

...
10 et0table
20 br0table

At the container config file (/var/lib/lxc/[container name]/config):

...
lxc.network.type = vets
lxc.network.link = br0
lxc.network.flags = up
lxc.network.hwadr = [auto create when bringing up container]
lxc.network.ipv4 = 195.x.x.4/24
lxc.network.ipv4.gateway = 195.x.x.1
lxc.network.veth.pair = [readable server name] (when using ifconfig)
lxc.start.auto = 0 (1 if you want the server to autostart)
lxc.start.delay = 0 (fill in seconds you want the container to wait before start)

I tested it by enabling apache2 on the container and accessed the webpage from outside the network. Hope it helps anybody who bumps into the same challenge I did.

PS: Do not forget if you choose to have the container's config file to assign the IP, that you disable it in the interface file of the container itself.

auto lo
iface lo inet loopback

auto eth0
iface eth0 net manual