CODEWITHSUNDEEP
javaclinuxvpniptables
ram
ram

Reputation: 1169

How to MASQUERADE tap interface traffic

I am currently developing a VPN server in Java, at least as much in Java as possible, and I am planning to perform routing of client packets through tap devices.

Currently, I am able to write ethernet frames to the tap device and I can observe these packets through tcpdump. However they are not routed over eth0, although I enabled ip forwarding and added a MASQUERADE rule to the iptables. (This problem seems identical with that, except that the gateway interface is a real interface there and a virtual interface in my situation.)

The output of ifconfig tap0 is as follows:

tap0      Link encap:Ethernet  HWaddr 82:7d:95:39:71:a1  
          inet addr:10.1.0.1  Bcast:10.1.255.255  Mask:255.255.0.0
          inet6 addr: fe80::807d:95ff:fe39:71a1/64 Scope:Link
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:767 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:56838 (56.8 KB)  TX bytes:0 (0.0 B)

The output of ip link show tap0 is as follows:

12: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 82:7d:95:39:71:a1 brd ff:ff:ff:ff:ff:ff

Here is how I attach to the tap device:

int helper_open(const char* dev_name, int tun_or_tap) {
    struct ifreq ifr;
    int fd;

    if ((fd = open("/dev/net/tun", O_RDWR)) == -1) {
        return -1;
    }

    memset(&ifr, 0, sizeof (ifr));
    strncpy(ifr.ifr_name, dev_name, IFNAMSIZ);
    ifr.ifr_flags = IFF_NO_PI;
    if (tun_or_tap == DEVICE_TUN) {
        ifr.ifr_flags |= IFF_TUN;
    } else if (tun_or_tap == DEVICE_TAP) {
        ifr.ifr_flags |= IFF_TAP;
    } else {
        return -2;
    }

    if (ioctl(fd, TUNSETIFF, (void *) &ifr) == -1) {
        close(fd);
        return -3;
    }

    return fd;
}

After successfully obtaining a file descriptor, it is trivial to write to the device through write() calls.

How I prepare the ethernet frame is as follows:

public boolean sendIp(byte[] buffer, int start, int length) {
    byte[] frame = new byte[length+14];
    System.arraycopy(mac, 0, frame, 0, 6);
    System.arraycopy(mac, 0, frame, 6, 2);
    byte[] ip = IpUtils.getSourceIp(buffer, start).getAddress();
    for (int i = 0; i < 4; i++) {
        frame[8+i] = (byte) (0xFF & (ip[i] ^ mac[i+2]));
    }
    frame[12] = 0x08;
    frame[13] = 0x00;
    System.arraycopy(buffer, start, frame, 14, length);
    try {
        write(frame, 0, frame.length);
        return true;
    } catch (IOException e) {
        logger.error("cannot send ip packet.", e);
        return false;
    }
}

mac is the MAC address of the tap device, and I generate the MAC address of the client by XORing the last four bytes of the tap device's MAC with the virtual IP that is assigned by me. (In my tests, the client's IP is 10.1.0.2.) In this way, it will be unique for all participants and also it will be easy to handle ARP/RARP protocols.

As it can be seen from RX packets field of ifconfig output, packets are received in tap device. Also, sample tcpdump -i tap0 -n output is as follows:

15:53:48.395082 IP 10.1.0.2.47132 > 216.58.208.34.443: Flags [S], seq 3162009985, win 65535, options [mss 1460,sackOK,TS val 4294939804 ecr 0,nop,wscale 6], length 0
15:53:49.396355 IP 10.1.0.2.39713 > 216.58.208.42.443: Flags [S], seq 2459164785, win 65535, options [mss 1460,sackOK,TS val 4294939905 ecr 0,nop,wscale 6], length 0
15:53:49.678691 IP 10.1.0.2.58306 > 194.177.210.54.123: NTPv3, Client, length 48
15:53:50.508132 IP 10.1.0.2.38112 > 172.217.22.110.443: Flags [S], seq 3132386571, win 65535, options [mss 1460,sackOK,TS val 4294940016 ecr 0,nop,wscale 6], length 0
15:53:51.519119 IP 10.1.0.2.37492 > 216.58.207.42.443: Flags [S], seq 3750738666, win 65535, options [mss 1460,sackOK,TS val 4294940117 ecr 0,nop,wscale 6], length 0

The packets are correctly decoded by tcpdump, so it seems I am preparing ethernet frames successfully. sysctl net.ipv4.ip_forward shows that ip forwarding is enabled. Then why are they not routed through eth0?

Output of iptables -L -n -v -t nat:

Chain PREROUTING (policy ACCEPT 2436 packets, 132K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 2436 packets, 132K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 20 packets, 1462 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   20  1462 MASQUERADE  all  --  *      eth0    0.0.0.0/0            0.0.0.0/0

and output of route -n:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         173.212.233.1   0.0.0.0         UG    0      0        0 eth0
10.1.0.0        0.0.0.0         255.255.0.0     U     0      0        0 tap0
173.212.233.0   173.212.233.1   255.255.255.0   UG    0      0        0 eth0
173.212.233.0   0.0.0.0         255.255.255.0   U     0      0        0 eth0

Any kind of help is highly appreciated.

P.S: I am developing on Ubuntu 16.04.

EDIT: To make sure that the packets do not depart from eth0, I started tcpdump -i eth0 host 5.189.147.197 -n and tcpdump -i tap0 host 5.189.147.197 -n on separate terminals and in the meanwhile the client attempted to connect to 5.189.147.197. I observed the traffic on tap0 interface but not on eth0 interface. So they are not forwarded, for sure.

Upvotes: 4

Views: 2341

Answers (1)

ram
ram

Reputation: 1169

As always, the reason was quite simple: While injecting the packets, I miscalculated the checksum in IP header. Namely, I forgot to flip the bits after the additions. Shame on me :(.

I am not deleting the whole question since it might help people while developing similar applications.

To checkout whether the checksums are OK, you can run tcpdump in a more verbose mode, i.e. including a few -v parameters. I used tcpdump -i tap0 -n -X -s 0 -vvv to also observe the content of the packets. It helped me a lot.

Regarding the bridging option suggested by user1794469, you need to have additional IPs at the subnet of eth0 that are available to assign to the clients, because bridging makes a client as if a new participant (without any NAT) joined to the subnet. However, my VPS has a single IP assigned to it, and it is not behind a NAT, so no DHCP etc. That's why I need to have my own NAT. Now it works flawlessly.

Upvotes: 5

Related Questions