passing a struct over TCP (SOCK_STREAM) socket in C

Question

I have a small client server application in which i wish to send an entire structure over a TCP socket in C not C++. Assume the struct to be the following:

    struct something{
int a;
char b[64];
float c;
}

I have found many posts saying that i need to use pragma pack or to serialize the data before sending and recieveing.

My question is, is it enough to use JUST pragma pack or just serialzation ? Or do i need to use both?

Also since serialzation is processor intensive process this makes your performance fall drastically, so what is the best way to serialize a struct WITHOUT using an external library(i would love a sample code/algo)?

Answer 1

Google Protocol Buffer offers a nifty solution to this problem. Refer here Google Protobol Buffer - C Implementaion

Create a .proto file based on the structure of your payload and save it as payload.proto

syntax="proto3"

message Payload {
     int32 age = 1;
     string name = 2;
} . 

Compile the .proto file using

protoc --c_out=. payload.proto

This will create the header file payload.pb-c.h and its corresponding payload.pb-c.c in your directory.

Create your server.c file and include the protobuf-c header files

#include<stdio.h>
#include"payload.pb.c.h"

int main()
{
   Payload pload = PLOAD__INIT;
   pload.name = "Adam";
   pload.age = 1300000;

   int len = payload__get_packed_size(&pload);

   uint8_t buffer[len];

   payload__pack(&pload, buffer);

   // Now send this buffer to the client via socket. 
}

On your receiving side client.c

....
int main()
{
   uint8_t buffer[MAX_SIZE]; // load this buffer with the socket data. 
   size_t buffer_len; // Length of the buffer obtain via read()
   Payload *pload = payload_unpack(NULL, buffer_len, buffer);

   printf("Age : %d Name : %s", pload->age, pload->name);
}

Make sure you compile your programs with -lprotobuf-c flag

gcc server.c payload.pb-c.c -lprotobuf-c -o server.out
gcc client.c payload.pb-c.c -lprotobuf-c -o client.out

Answer 2

You need the following to portably send struct's over the network:

• Pack the structure. For gcc and compatible compilers, do this with __attribute__((packed)).

• Do not use any members other than unsigned integers of fixed size, other packed structures satisfying these requirements, or arrays of any of the former. Signed integers are OK too, unless your machine doesn't use a two's complement representation.

• Decide whether your protocol will use little- or big-endian encoding of integers. Make conversions when reading and writing those integers.

• Also, do not take pointers of members of a packed structure, except to those with size 1 or other nested packed structures. See this answer.

A simple example of encoding and decoding follows. It assumes that the byte order conversion functions hton8(), ntoh8(), hton32(), and ntoh32() are available (the former two are a no-op, but there for consistency).

#include <stdint.h>
#include <inttypes.h>
#include <stdlib.h>
#include <stdio.h>

// get byte order conversion functions
#include "byteorder.h"

struct packet {
    uint8_t x;
    uint32_t y;
} __attribute__((packed));

static void decode_packet (uint8_t *recv_data, size_t recv_len)
{
    // check size
    if (recv_len < sizeof(struct packet)) {
        fprintf(stderr, "received too little!");
        return;
    }

    // make pointer
    struct packet *recv_packet = (struct packet *)recv_data;

    // fix byte order
    uint8_t x = ntoh8(recv_packet->x);
    uint32_t y = ntoh32(recv_packet->y);

    printf("Decoded: x=%"PRIu8" y=%"PRIu32"\n", x, y);
}

int main (int argc, char *argv[])
{
    // build packet
    struct packet p;
    p.x = hton8(17);
    p.y = hton32(2924);

    // send packet over link....
    // on the other end, get some data (recv_data, recv_len) to decode:
    uint8_t *recv_data = (uint8_t *)&p;
    size_t recv_len = sizeof(p);

    // now decode
    decode_packet(recv_data, recv_len);

    return 0;
}

As far as byte order conversion functions are concerned, your system's htons()/ntohs() and htonl()/ntohl() can be used, for 16- and 32-bit integers, respectively, to convert to/from big-endian. However, I'm not aware of any standard function for 64-bit integers, or to convert to/from little endian. You can use my byte order conversion functions; if you do so, you have to tell it your machine's byte order by defining BADVPN_LITTLE_ENDIAN or BADVPN_BIG_ENDIAN.

As far as signed integers are concerned, the conversion functions can be implemented safely in the same way as the ones I wrote and linked (swapping bytes directly); just change unsigned to signed.

UPDATE: if you want an efficient binary protocol, but don't like fiddling with the bytes, you can try something like Protocol Buffers (C implementation). This allows you to describe the format of your messages in separate files, and generates source code that you use to encode and decode messages of the format you specify. I also implemented something similar myself, but greatly simplified; see my BProto generator and some examples (look in .bproto files, and addr.h for usage example).

Answer 3

Usually, serialization brings several benefits over e.g. sending the bits of the structure over the wire (with e.g. fwrite).

1. It happens individually for each non-aggregate atomic data (e.g. int).
2. It defines precisely the serial data format sent over the wire
3. So it deals with heterogenous architecture: sending and recieving machines could have different word length and endianness.
4. It may be less brittle when the type change a little bit. So if one machine has an old version of your code running, it might be able to talk with a machine with a more recent version, e.g. one having a char b[80]; instead of char b[64];
5. It may deal with more complex data structures -variable-sized vectors, or even hash-tables- with a logical way (for the hash-table, transmit the association, ..)

Very often, the serialization routines are generated. Even 20 years ago, RPCXDR already existed for that purpose, and XDR serialization primitives are still in many libc.

Answer 4

It depends on whether you can be sure that your systems on either end of the connection are homogeneous or not. If you are sure, for all time (which most of us cannot be), then you can take some shortcuts - but you must be aware that they are shortcuts.

struct something some;
...
if ((nbytes = write(sockfd, &some, sizeof(some)) != sizeof(some))
    ...short write or erroneous write...

and the analogous read().

However, if there's any chance that the systems might be different, then you need to establish how the data will be transferred formally. You might well linearize (serialize) the data - possibly fancily with something like ASN.1 or probably more simply with a format that can be reread easily. For that, text is often beneficial - it is easier to debug when you can see what's going wrong. Failing that, you need to define the byte order in which an int is transferred and make sure that the transfer follows that order, and the string probably gets a byte count followed by the appropriate amount of data (consider whether to transfer a terminal null or not), and then some representation of the float. This is more fiddly. It is not all that hard to write serialization and deserialization functions to handle the formatting. The tricky part is designing (deciding on) the protocol.

Answer 5

Pragma pack is used for the binary compatibility of you struct on another end. Because the server or the client to which you send the struct may be written on another language or builded with other c compiler or with other c compiler options.

Serialization, as I understand, is making stream of bytes from you struct. When you write you struct in the socket you make serialiazation.

Answer 6

Why would you do this when there are good and fast serialization libraries out there like Message Pack which do all the hard work for you, and as a bonus they provide you with cross-language compatibility of your socket protocol?

Use Message Pack or some other serialization library to do this.

Answer 7

Before you send any data over a TCP connection, work out a protocol specification. It doesn't have to be a multiple-page document filled with technical jargon. But it does have to specify who transmits what when and it must specify all messages at the byte level. It should specify how the ends of messages are established, whether there are any timeouts and who imposes them, and so on.

Without a specification, it's easy to ask questions that are simply impossible to answer. If something goes wrong, which end is at fault? With a specification, the end that didn't follow the specification is at fault. (And if both ends follow the specification and it still doesn't work, the specification is at fault.)

Once you have a specification, it's much easier to answer questions about how one end or the other should be designed.

I also strongly recommend not designing a network protocol around the specifics of your hardware. At least, not without a proven performance issue.

Answer 8

You could use an union with the structure you want to send and an array:

union SendSomething {
    char arr[sizeof(struct something)];
    struct something smth;
};

This way you can send and receive just arr. Of course, you have to take care about endianess issues and sizeof(struct something) might vary across machines (but you can easily overcome this with a #pragma pack).