macOS NKE ipf_filter – IP checksum is 0

Question

I’m intercepting, modifying and re-injecting outgoing IPv4 TCP packets via an IP filter. The problem is that, after I alter the packet and set the IP and TCP checksums, when I analyse the resulting packet with Wireshark, the IP checksum equals 0 (the checksum I’m calculating seems to be correct since it’s equal to the Wireshark’s suggested one).

Here’s the procedure I’m following, I hope there's someone who can spot the error or suggest a better way of handling things:

    static int handle_packet(mbuf_t* data, int ip_len, int dir, ipf_pktopts_t options)
{
    errno_t result = 0;
    unsigned char packet[1500];
    struct tcphdr *tcp;
    struct ip *ip;
    mbuf_t old_packet = *data, new_packet;
    uint32_t mbufs = 0, packet_bytes = 0;

    // zero packet
    bzero(packet, sizeof(packet));

    // "finalize" the packet so that it is safe to modify it
    mbuf_outbound_finalize(*data, AF_INET, 0);

    // get length of mbuf chain
    do
    {
        mbufs++;
        packet_bytes += mbuf_len(old_packet);
        old_packet = mbuf_next(old_packet);
    } while (old_packet != NULL);

    // copy data to local buffer
    if (0 != (result = mbuf_copydata(*data, 0, packet_bytes, packet))) {
        printf("mbuf_copydata returned %d", result);
        return 0;
    }

    // pointer to start IP header
    ip = (struct ip*)packet;
    tcp = (struct tcphdr*)((u_int32_t*)ip + ip->ip_hl);

    // only consider SYN packet
    if (!(tcp->th_flags & TH_SYN))
        return KERN_SUCCESS;

    if (0 != (result = mbuf_dup(*data, MBUF_DONTWAIT, &new_packet)))
    {
        printf("ERROR - mbuf_dup: unable to duplicate mbuf, %d", result);
        return 0;
    }


/**
… I’m modifying the packet and recalculating ip and tcp’s checksums here
(by previously setting them to 0, so to avoid that the previous values
are considered in the calculation) …
*/


    /*
     * Copy buffer back to mbuf
     */
    if (0 != (result = mbuf_copyback(new_packet, 0, ntohs(ip->ip_len), packet, MBUF_DONTWAIT)))
    {
         mbuf_freem(new_packet);

        switch (result) {
            case EINVAL:
                printf("ERROR - handle_packet: mbuf_copyback returned EINVAL");
                return 0;
                break;
            case ENOBUFS:
                printf("ERROR - handle_packet: mbuf_copyback returned ENOBUFS");
                return 0;
                break;
            default:
                break;
        }
    }

    // recompute any checksums invalidated by data changes
//    mbuf_outbound_finalize(new_packet, AF_INET, 0); // -> PANIC(m->m_flags & M_PKTHDR)

  // is this necessary?
    mbuf_set_csum_performed(new_packet, MBUF_CSUM_DID_IP | MBUF_CSUM_IP_GOOD | MBUF_CSUM_DID_DATA | MBUF_CSUM_PSEUDO_HDR, checksum_ip(ip));

    result = ipf_inject_output(new_packet, ip_filter_ref, options);

    return result == 0 ? EJUSTRETURN : result;
}

static errno_t ip_filter_output(void* cookie, mbuf_t *data, ipf_pktopts_t options)
{
    struct ip *ip;
    char src[32], dst[32];
    int ip_len;

    // pointer to start IP header
    ip = (struct ip*)mbuf_data(*data);
    ip_len = ntohs(ip->ip_len);

    bzero(src, sizeof(src));
    bzero(dst, sizeof(dst));

    // converts the network address structure into a character string
    inet_ntop(AF_INET, &ip->ip_src, src, sizeof(src));
    inet_ntop(AF_INET, &ip->ip_dst, dst, sizeof(dst));

    // avoid congestion and filter only packets from/to tcpcrypt website
    if (ip->ip_p == IPPROTO_TCP
        && mbuf_flags(*data) == MBUF_PKTHDR) {
        return handle_packet(data, ip_len, DIRECTION_OUT /* 1 */, options);
    }

    // continue with normal processing of the packet
    return KERN_SUCCESS;
}

I suspect that getting zero is the expected consequence of recalculating the checksum over the entire header, i.e. it causes the calculated sum to cancel itself out.

Anyway, I really cannot understand why this is happening.

Does anyone know the answer or can help?

Thank you very much in advance,

Romeo

Answer 1

First of all, this

do
{
    mbufs++;
    packet_bytes += mbuf_len(old_packet);
    old_packet = mbuf_next(old_packet);
} while (old_packet != NULL);

is a horrible way of getting the length of a mbuf chain. The correct way is

size_t totalLength = (
    mbuf_flags(mbuf) & MBUF_PKTHDR ?
    mbuf_pkthdr_len(mbuf) : mbuf_len(mbuf)
);

As either it is a chain of mbufs but then the first mbuf of that chain is supposed to have a paket header and this paket header contains the size of the entire mbuf chain (a chain without such a header is broken by definition) or it's just a single mbuf but then it is enough to just ask this one mbuf for its size. Throughout the entire kernel code, the size of an mbuf chain is retrieved as shown by my code above.

Then please understand what mbuf_outbound_finalize() does. From the documentation of that method:

This function will "finalize" the packet allowing your code to inspect the final packet.

There are a number of operations that are performed in hardware, such as calculating checksums. This function will perform in software the various opterations that were scheduled to be done in hardware.

So it won't matter if checksum calculation is offloaded to hardware or not, once you call that function, the packets are supposed to have correct checksums. If you then modify the packets, it's up to you to fix the checksums again. You can do this by recalculating the checksums from scratch, once your are done with modifications, or by "fixing" the existing ones (e.g. if you know which old data turned into which new data, it's possible to "fix" the old checksum to match the new data w/o calculating everything from ground up, it's a bit tricky but makes calculations much faster).

But all of this cannot be done by just calling mbuf_outbound_finalize() again in the end. You can call this function exactly once on an mbuf or mbuf chain and after that it's up to you, that means your code to keep checksums correct. mbuf_outbound_finalize() only does something if checksum calculation was not done yet and has been scheduled to be done in hardware. If your hardware doesn't support offloading, then the packet will already have a correct checksum once your filter catches it and then mbuf_outbound_finalize() will do nothing as there is nothing left for it to do.

Final issue: You are not supposed to call mbuf_set_csum_performed(). Please see the documentation of that function:

This is used by the driver to indicate to the stack which checksum operations were performed in hardware.

"by the driver" and you are not the driver; despite that this function is used for incoming and not for outgoing packets.

Answer 2

Are you seeing non-zero checksums if you don't run your program?

If not, it's possible that your NIC does checksum offloading, which makes the driver not actually insert any checksum into the packet (as the network interface will take care of it).

See section 7.10.2. (Checksum offloading) of the WireShark documentation.