NachFenix
NachFenix

Reputation: 11

Kafka CRC calculation

I´m trying to generate a kafka producer message by native tcpip on .net, I have the message already done, but I´m missing something on CRC calculation, I generate a CRC calculation with:

  public class CRC32C
{
    private const uint POLY = 0x1EDC6F41;
    private static readonly uint[,] crc32cTable = new uint[8, 256];

    // Function to initialize the CRC32C table
    public void InitCRC32C()
    {
        uint n, crc, k;

        // Initialize the CRC32C table
        for (n = 0; n < 256; n++)
        {
            crc = n;
            for (int i = 0; i < 8; i++)
            {
                crc = (crc & 1) != 0 ? (crc >> 1) ^ POLY : crc >> 1;
            }
            crc32cTable[0, n] = crc;
        }

        for (n = 0; n < 256; n++)
        {
            crc = crc32cTable[0, n];
            for (k = 1; k < 8; k++)
            {
                crc = crc32cTable[0, crc & 0xff] ^ (crc >> 8);
                crc32cTable[k, n] = crc;
            }
        }
    }

    // Software CRC32C calculation
    public uint CRC32Ccalculate(uint crc, byte[] buffer, int length)
    {
        uint crcVal = crc ^ 0xffffffff;
        int index = 0;

        // Process any leading bytes to bring the data pointer to an eight-byte boundary
        while (length > 0 && (index & 7) != 0)
        {
            crcVal = crc32cTable[0, (crcVal ^ buffer[index]) & 0xff] ^ (crcVal >> 8);
            index++;
            length--;
        }

        // Process 8-byte aligned blocks
        while (length >= 8)
        {
            ulong ncopy = BitConverter.ToUInt64(buffer, index);
            crcVal ^= (uint)ncopy; // Assume the buffer is little-endian
            crcVal = crc32cTable[7, crcVal & 0xff] ^
                     crc32cTable[6, (crcVal >> 8) & 0xff] ^
                     crc32cTable[5, (crcVal >> 16) & 0xff] ^
                     crc32cTable[4, (crcVal >> 24) & 0xff] ^
                     crc32cTable[3, (crcVal >> 32) & 0xff] ^
                     crc32cTable[2, (crcVal >> 40) & 0xff] ^
                     crc32cTable[1, (crcVal >> 48) & 0xff] ^
                     crc32cTable[0, crcVal >> 56];
            index += 8;
            length -= 8;
        }

        // Process remaining bytes
        while (length > 0)
        {
            crcVal = crc32cTable[0, (crcVal ^ buffer[index]) & 0xff] ^ (crcVal >> 8);
            index++;
            length--;
        }

        return crcVal ^ 0xffffffff;
    }


}

Here it´s correct message from Python producer: Python message generated 0000008d000000070000000300176b61666b612d707974686f6e2d70726f64756365722d32ffff0001000075300000000100086d792d746f70696300000001000000000000004a00000000000000000000003e00000000021d161b720000000000000000019401ee89d10000019401ee89d1ffffffffffffffffffffffffffff000000011800000006666f6f0662617200 where 1d161b72 its the ACK. I tried to make calculation, but I cannot reach this value. Seems i should take the values from CRC in advance to the end of message, but not get same result. I suppose Algorithm it´s not correct, but i´m not sure about it.

So when I try to generate the message by my self, I get error on the kafka response as -2 error, mean CRC calculation error. Any help on this? My target it's implement Kafka producer over microcontroller protocol, but I would like to test before on .net

Kafka message over wireshark

Upvotes: 0

Views: 53

Answers (0)

Related Questions