Linux IP forwarding doesn't work

Question

I ran into this weird issue. The same setup worked before, but suddenly stopped. Two machines [Internet] <-WAN-> Gateway <-VLAN-> Core

I am trying to route Core via Gateway. Core:

root@core:~# ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 76:61:6b:7a:65:af
          inet addr:10.0.0.2  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::7461:6bff:fe7a:65af/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:38423 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3814 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1959037 (1.9 MB)  TX bytes:501771 (501.7 KB)

root@core:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.0.1        0.0.0.0         UG    0      0        0 eth1
10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth1

root@core:~# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=48 time=10.6 ms
^C
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 10.644/10.644/10.644/0.000 ms

Gateway:

root@gateway:~# cat /proc/sys/net/ipv4/ip_forward
1
root@gateway:~# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 3e:50:8a:be:b9:80
          inet addr:83.222.241.213  Bcast:83.222.241.255  Mask:255.255.255.0
          inet6 addr: fe80::3c50:8aff:febe:b980/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4536 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4197 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:485439 (485.4 KB)  TX bytes:798131 (798.1 KB)

root@gateway:~# ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 42:50:8a:be:b9:80
          inet addr:10.0.0.1  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::4050:8aff:febe:b980/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1985 errors:0 dropped:0 overruns:0 frame:0
          TX packets:13169 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:256280 (256.2 KB)  TX bytes:701930 (701.9 KB)

root@gateway:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         83.222.241.1    0.0.0.0         UG    0      0        0 eth0
10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth1
83.222.241.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
root@gateway:~# iptables-save
# Generated by iptables-save v1.4.21 on Thu Oct 23 23:13:32 2014
*nat
:PREROUTING ACCEPT [3:180]
:INPUT ACCEPT [3:180]
:OUTPUT ACCEPT [173:10388]
:POSTROUTING ACCEPT [170:10200]
-A POSTROUTING -o eth0 -j MASQUERADE
COMMIT
# Completed on Thu Oct 23 23:13:32 2014
# Generated by iptables-save v1.4.21 on Thu Oct 23 23:13:32 2014
*filter
:INPUT ACCEPT [581:49229]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [819:73373]
-A FORWARD -i eth1 -j ACCEPT
COMMIT
# Completed on Thu Oct 23 23:13:32 2014
# Generated by iptables-save v1.4.21 on Thu Oct 23 23:13:32 2014
*mangle
:PREROUTING ACCEPT [581:49229]
:INPUT ACCEPT [581:49229]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [822:73737]
:POSTROUTING ACCEPT [822:73737]
COMMIT
# Completed on Thu Oct 23 23:13:32 2014
# Generated by iptables-save v1.4.21 on Thu Oct 23 23:13:32 2014
*raw
:PREROUTING ACCEPT [581:49229]
:OUTPUT ACCEPT [822:73737]
COMMIT
# Completed on Thu Oct 23 23:13:32 2014

Now running tcpdump -i any -n -v udp and port 53 on Gateway I run:

root@gateway:~# dig test.com @8.8.8.8 &>/dev/null
23:16:33.426336 IP (tos 0x0, ttl 64, id 16201, offset 0, flags [none], proto UDP (17), length 65)
    83.222.241.213.41376 > 8.8.8.8.53: 39515+ [1au] A? test.com. (37)
23:16:33.436145 IP (tos 0x0, ttl 49, id 26701, offset 0, flags [none], proto UDP (17), length 81)
    8.8.8.8.53 > 83.222.241.213.41376: 39515 1/0/1 test.com. A 50.23.225.49 (53)

If I run dig on Core (capture from Gateway):

23:17:55.801448 IP (tos 0x0, ttl 64, id 50634, offset 0, flags [none], proto UDP (17), length 65)
    10.0.0.2.55008 > 8.8.8.8.53: 6910+ [1au] A? test.com. (37)
23:17:55.801539 IP (tos 0x0, ttl 63, id 50634, offset 0, flags [none], proto UDP (17), length 65)
    83.222.241.213.55008 > 8.8.8.8.53: 6910+ [1au] A? test.com. (37)
23:18:00.801477 IP (tos 0x0, ttl 64, id 50635, offset 0, flags [none], proto UDP (17), length 65)
    10.0.0.2.55008 > 8.8.8.8.53: 6910+ [1au] A? test.com. (37)
23:18:00.801559 IP (tos 0x0, ttl 63, id 50635, offset 0, flags [none], proto UDP (17), length 65)
    83.222.241.213.55008 > 8.8.8.8.53: 6910+ [1au] A? test.com. (37)
23:18:05.801862 IP (tos 0x0, ttl 64, id 50636, offset 0, flags [none], proto UDP (17), length 65)
    10.0.0.2.55008 > 8.8.8.8.53: 6910+ [1au] A? test.com. (37)
23:18:05.801932 IP (tos 0x0, ttl 63, id 50636, offset 0, flags [none], proto UDP (17), length 65)
    83.222.241.213.55008 > 8.8.8.8.53: 6910+ [1au] A? test.com. (37)

So somehow my pings get delivered, but my UDP packets don't? I guess the ip_forward is not rewriting the source address somehow? Any ideas?

Answer 1

So the problem was with VirtIO network cards in QEMU.

When two VirtIO VMs are running on the same physical machine, the one behind the NAT starts sending packets with bad checksums for some strange reason. Switching off checksum checks solves the issue:

ethtool -K eth1 tx off tso off ufo off gso off

And to make sure it persists over reboots:

echo "ethtool -K eth1 tx off tso off ufo off gso off" >> /etc/rc.local

Credit goes to my VPS support.