How to guarantee that write() will transfer entire int over socket?

Question

First of all I found similar question here. But I didn't understand how this should work:

int data = rand();
char* tosend = (char*)&data;
int remaining = sizeof(data);

I want to send size of file which I store into int variable. And I use next code for this:

Server side:

struct stat st;
stat("file", &st);
int file_size = st.st_size;
write(client_fd, &file_size, sizeof(file_size));

Client side:

int file_size;
read(server_fd, &file_size, sizeof(int));

And it works fine. But in documentation of write says that:

Note that a successful write() may transfer fewer than count bytes

And same for read:

It is not an error if this number is smaller than the number of bytes requested

If I'd send buffer of chars, which are 1 byte long each. I would just add offset to the buffer and store each next of bytes in buffer. Like this:

int write_all(int fd, void* buffer, int count) {
    int offset = 0, b = 0, status;

    while ((status = write(fd, buffer + offset, count)) != count) {
        if (status == -1) {
            /error check/
            return -1;
        }

        count -= status;
        offset += status;
        b += status;
    }

    b += status;

    return b;
}

But how to make the same thing for variable?

Answer 1

Your write_all() implementation is faulty. It will fail on short writes.

Consider these:

size_t write_some(const int desc, const void *buffer, const size_t min_length, const size_t max_length)
{
    const char       *ptr  = (const char *)buffer;
    const char *const need = (const char *)buffer + min_length;
    const char *const end  = (const char *)buffer + max_length;

    while (ptr < need) {
        ssize_t  len = write(desc, ptr, (size_t)(end - ptr));
        if (len > 0) {
            ptr += len;
        } else
        if (len != -1) {
            errno = EIO;
            return (size_t)(ptr - (const char *)buffer);
        } else {
            /* errno set by write() */
            return (size_t)(ptr - (const char *)buffer);
        }
    }

    errno = 0;
    return (size_t)(ptr - (const char *)buffer);
}

size_t  read_some(const int desc, void *buffer, const size_t min_length, const size_t max_length)
{
    char *const end  = (char *)buffer + max_length;
    char *const need = (char *)buffer + min_length;
    char       *ptr  = (char *)buffer;

    while (ptr < need) {
        ssize_t  len = read(desc, ptr, (size_t)(end - ptr));
        if (len > 0) {
            ptr += len;
        } else
        if (len == 0) {
            errno = 0;
            return (size_t)(ptr - (char *)buffer);
        } else
        if (len != -1) {
            errno = EIO;
            return (size_t)(ptr - (char *)buffer);
        } else {
            /* errno set by read() */
            return (size_t)(ptr - (char *)buffer);
        }
    }

    errno = 0;
    return (size_t)(ptr - (char *)buffer);
}

They both always set errno, to zero if successful, errno otherwise, and return the actual number of bytes sent/received.

If you receive a stream (using descriptor desc) of packets starting with four-byte big-endian (network-endian) length fields, the length including the four-byte length field itself (so that its minimum valid value is 4), you can use the following:

static inline size_t packet_size4(const void *const from)
{
    const unsigned char *src = (const unsigned char *)from;
    return ((size_t)(src[0]) << 24)
         | ((size_t)(src[1]) << 16)
         | ((size_t)(src[2]) <<  8)
         |           src[3];
}

int handle_all_packets(int desc, unsigned char *buffer, size_t maxlen)
{
    size_t  have = 0;
    size_t  len = 0;

    while (1) {
        if (have < 4) {
            have += read_some(desc, buffer + have, 4 - have, maxlen - have);
            if (have < 4)
                return errno;
            len = packet_size4(buffer);
            if (len < 4)
                return errno = EBADMSG;
            if (len > maxlen)
                return errno = EOVERFLOW;
        }
        if (have < len) {
            have += read_some(desc, buffer + have, len - have, maxlen - have);
            if (errno)
                return errno;
            if (have < len)
                return errno = EIO;
        }

        /* Process len-byte packet in buffer. */

        if (have > len) {
            memmove(buffer, buffer + len, have - len);
            have -= len;
            len   = 0;
        } else {
            have  = 0;
            len   = 0;
        }
    }
}

If there were no errors, it will return 0, otherwise it returns a nonzero errno error code.

To send such a packet, you can use the following:

static inline void set_packet_size4(unsigned char *dst, size_t size)
{
    dst[0] = (size >> 24) & 255;
    dst[1] = (size >> 16) & 255;
    dst[2] = (size >>  8) & 255;
    dst[3] =  size        & 255;
}

int send_packet(const int desc, const void *data, const size_t len)
{
    size_t  n;

    {
        unsigned char  buf[4];
        set_packet_size4(buf, len + 4);
        n = write_some(desc, buf, 4, 4);
        if (errno)
            return errno;
        if (n != 4)
            return errno = EIO;
    }

    n = write_some(desc, data, len, len);
    if (errno)
        return errno;
    if (n != len)
        return errno = EIO;
    return 0;
}

If the packet is written correctly, the function returns 0, otherwise it returns a nonzero errno error code.

Answer 2

Both write and read return the number of bytes sent or received respectively. Don't start processing your data before you have received enough bytes to do so.

In more detail: TCP/IP will try to pass on the data it has as quickly as possible. Therefore if data is split between packets, your read may return before the expected number of bytes has been received, this may allow you to get a head start on data processing instead of waiting.

Answer 3

There is actually a standardized network byte order (which happens to be big endian), and standard functions to convert from host to network and vice versa:

• htons() Host to Network Short
• htonl() Host to Network Long
• ntohl() Network to Host Long
• ntohs() Network to Host Short

Use these for a portable way to transfer binary integers over the network. You might also look at writen() for really, really trying to write n bytes, although that function comes with its own set of pitfalls.

Answer 4

The example you refer to is not great because it gives no consideration of the integer byte order - and you have no guarantee that the byte order on the receiving host will be the same as on the host you are sending from.

My suggestion would be to

1. choose a byte order - LSB first (little endian) or MSB first (big endian), and then
2. allocate a char or uint8_t buffer of the relevant size (e.g. 2 bytes for a 16-bit integer, 4 bytes for a 32-bit integer) and store the integer there byte by byte
3. use your existing write_all() function to send it.

Big-endian is maybe more typical for network data, so it might be better to use that. Then, if you want to send a 32-bit integer ...

void send_u32(int client_fd, int32_t n) {
  uint8_t buf[4];

  // Big-endian byte order ...
  for (int i=0; i<4; i++) {
      buf[i] = n >> ((3-i)*8);
  }

  int res = write_all(client_fd, buf, 4);
}

For 16-bit integer just use a 2-byte buffer.

You will need to #include <stdint.h> for above.

P.S. st_size in struct stat is of type off_t - which I think might be OS-dependent. Maybe someone who knows more about POSIX can shed some light on that. (You can probably forget about using a 16-bit integer anyway - as I expect it will always be at least a 32-bit integer, except maybe on some embedded systems?.)