Receive continuous stream of varying length packets over sockets in C at a quick rate?

Question

I'm working on the sockets (in C, with no prior experience on socket programming) from the past few days. Actually I have to collect WiFi packets on raspberry pi, do some processing and have to send the formatted information to another device over sockets (both the devices are connect in a network).

The challenge I'm facing is when receiving the data over the sockets.

While sending the data, the data is sent successfully over the sockets from the sending side but on the receiving side, sometimes some junk or previous data is received.

On Sending Side (client):

int server_socket = socket(AF_INET, SOCK_STREAM, 0);
//connecting to the server with connect function
send(server_socket, &datalength, sizeof(datalength),0);  //datalength is an integer containing the number of bytes that are going to be sent next
send(server_socket, actual_data, sizeof(actual_data),0); //actual data is a char array containing the actual character string data

On Receiving Side (Server Side):

int server_socket = socket(AF_INET, SOCK_STREAM, 0);
//bind the socket to the ip and port with bind function
//listen to the socket for any clients
//int client_socket = accept(server_socket, NULL, NULL);
int bytes;
recv(client_socket, &bytes, sizeof(bytes),0);
char* actual_message = malloc(bytes);
int rec_bytes = recv(client_socket, actual_message, bytes,0);

*The above lines of code are not the actual lines of code, but the flow and procedure would be similar (with exception handling and comments).

Sometimes, I could get the actual data for all the packets quickly(without any errors and packet loss). But sometimes the bytes (integer sent to tell the size of byte stream for the next transaction) is received as a junk value, so my code is breaking at that point. Also sometimes the number of bytes that I receive on the receving side are less than the number of bytes expected (known from the received integer bytes). So in that case, I check for that condition and retrieve the remaining bytes.

Actually the rate at which packets arrive is very high (around 1000 packets in less than a second and I have to dissect, format and send it over sockets). I'm trying different ideas (using SOCK_DGRAMS but there is some packet loss here, insert some delay between transactions, open and close a new socket for each packet, adding an acknowledgement after receiving packet) but none are them meets my requirement (quick transfer of packets with 0 packet loss).

Kindly, suggest a way to send and receive varying length of packets at a quick rate over sockets.

Answer 1

I see a few main issues:

1. I think your code ignores the possibility of a full buffer in the send function.

   It also seems to me that your code ignores the possibility of partial data being collected by recv. (nevermind, I just saw the new comment on that)

   In other words, you need to manage a user-land buffer for send and handle fragmentation in recv.

2. The code uses sizeof(int) which might be a different length on different machines (maybe use uint32_t instead?).

3. The code doesn't translate to and from network byte order. This means that you're sending the memory structure of the int instead of an integer that can be read by different machines (some machines store the bytes backwards, some forward, some mix and match).

Notice that when you send larger data using TCP/IP, it will be fragmented into smaller packets.

This depends, among others, on the MTU network value (which often runs at ~500 bytes in the wild and usually around ~1500 bytes in your home network).

To handle these cases you should probably use an evented network design rather than blocking sockets.

Consider routing the send through something similar to this (if you're going to use blocking sockets):

int send_complete(int fd, void * data, size_t len) {
    size_t act = 0;
    while(act < len) {
        int tmp = send(fd, (void *)((uintptr_t)data + act), len - act);
        if(tmp <= 0 && errno != EWOULDBLOCK && errno != EAGAIN && errno != EINTR) 
            return tmp; // connection error
        act += tmp;
        // add `select` to poll the socket
    }
    return (int)act;
}

As for the sizeof issues, I would replace the int with a specific byte length integer type, such as int32_t.

A few more details

Please notice that sending the integer separately doesn't guaranty that it would be received separately or that the integer itself wouldn't be fragmented.

The send function writes to the system's buffer for the socket, not to the network (just like recv reads from the available buffer and not from the wire).

You can't control where fragmentation occurs or how the TCP packets are packed (unless you implement your own TCP/IP stack).

I'm sure it's clear to you that the "junk" value is data that was sent by the server. This means that the code isn't reading the integer you send, but reading another piece of data.

It's probably a question of alignment to the message boundaries, caused by an incomplete read or an incomplete send.

P.S.

I would consider using the Websocket protocol on top of the TCP/IP layer.

This guaranties a binary packet header that works with different CPU architectures (endianness) as well as offers a wider variety of client connectivity (such as connecting with a browser etc').

It will also solve the packet alignment issue you're experiencing (not because it won't exist, but because it was resolved in whatever Websocket parser you will adopt).