Does send() always send whole buffer?

Question

send() shall return the number of bytes sent or error code, but all examples that I found check it only with error codes, but not with the number of bytes sent.

//typical example
int cnt=send(s,query,strlen(query),0);
if (cnt < 0) return(NULL);
//Hey, what about cnt < strlen(query)?

Answer 1

I've read through this question and other two related questions:

When a non-blocking send() only transfers partial data, can we assume it would return EWOULDBLOCK the next call?

Blocking sockets: when, exactly, does "send()" return?

I found not all answers reach an consensus and one or two answers have oppsite conclusion.

So I spent quite some time searching in book and playing with this code of @Damon that he posted in the comment of https://stackoverflow.com/a/19400029/5983841 .

---

I think most answers are wrong and my conlusion is:

A call to send has these possible outcomes:

1. There is at least one byte available in the send buffer:
   1.1 → if send is blocking (the fd is not set as non-blocking and MSG_DONTWAIT is not specified in send), send blocks until there's enough room for the whole buffer to fit, and send the whole buffer.
   1.2 → if send is non-blocking (fd set as non-blocking or MSG_DONTWAIT is specified in send), send returns the number of bytes accepted (possibly fewer than you asked for).

2. The send buffer is completely full at the time you call send.
   → if the socket is blocking, send blocks
   → if the socket is non-blocking, send fails with EWOULDBLOCK/EAGAIN

3. An error occurred (e.g. user pulled network cable, connection reset by peer) →send fails with another error

#1.1 conforms to man 2 send:

When the message does not fit into the send buffer of the socket, send() normally blocks, unless the socket has been placed in nonblocking I/O mode.

partial recv is easy to understand, while for partial send (from The Linux Programming Interface):

61.1 Partial Reads and Writes on Stream Sockets

...

A partial write may occur if there is insufficient buffer space to transfer all of the requested bytes and one of the following is true:

• A signal handler interrupted the write() call (Section 21.5) after it transferred some of the requested bytes.
• The socket was operating in nonblocking mode (O_NONBLOCK), and it was possible to transfer only some of the requested bytes.
• An asynchronous error occurred after only some of the requested bytes had been transferred. By an asynchronous error, we mean an error that occurs asynchronously with respect to the application’s use of calls in the sockets API. An asynchronous error can arise, for example, because of a problem with a TCP connection, perhaps resulting from a crash by the peer application.

In all of the above cases, assuming that there was space to transfer at least 1 byte, the write() is successful, and returns the number of bytes that were transferred to the output buffer.

...

(The case of signal interruption doesn't happen in most of the time and I have difficulties writing to prove a partial write in this case. Hope someone could help)

What's not made clear enough of man 2 send :

When the message does not fit into the send buffer of the socket, send() normally blocks, unless the socket has been placed in nonblocking I/O mode. In nonblocking mode it would fail with the error EAGAIN or EWOULDBLOCK in this case.

is that in nonblocking mode it would fail if the buffer is completely full. If there's 1 byte available in send buffer, it won't fail but instead returns the number of bytes that were sent, aka a partial send. (the author of the book is also the mantainer of linux manpage https://www.kernel.org/doc/man-pages/maintaining.html ).

---

Prove of code, written by @Damon. I modifed 3~5 lines, making the server doesn't consume any packets, so as to demonstrate.

#include <cstdio>
#include <cstdlib>

#include <unistd.h>
#include <errno.h>

#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>

#include <netdb.h>
#include <arpa/inet.h>

int create_socket(bool server = false)
{
    addrinfo  hints = {};
    addrinfo* servinfo;

    int sockfd = -1;
    int rv;

    hints.ai_family   = AF_INET;
    hints.ai_socktype = SOCK_STREAM;
    hints.ai_flags    = server ? AI_PASSIVE : 0;

    if ((rv = getaddrinfo(server ? 0 : "localhost", "12345", &hints, &servinfo)))
    {
        printf("getaddrinfo failed: %s\n", gai_strerror(rv));
        exit(1);
    }

    for(auto p = servinfo; p; p = p->ai_next)
    {
        if ((sockfd = socket(p->ai_family, p->ai_socktype, p->ai_protocol)) == -1)
        {
            perror("socket");
            continue;
        }

        if(server)
        {
            int yes = 1;
            setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(yes));

            if(bind(sockfd, p->ai_addr, p->ai_addrlen) == -1)
            {
                close(sockfd);
                perror("bind");
                continue;
            }
        }
        else
        {
            if(connect(sockfd, p->ai_addr, p->ai_addrlen) == -1)
            {
                close(sockfd);
                perror("connect");
                continue;
            }
            else
                puts("client: connected");
        }

        break;
    }

    freeaddrinfo(servinfo);
    return sockfd;
}

void server()
{
    int socket = create_socket(true);
    if(listen(socket, 5) == -1)
    {
        perror("listen");
        exit(1);
    }
    puts("server: listening");

    int conn = -1;
    sockaddr_storage addr;
    unsigned int sizeof_addr = sizeof(addr);

    for(;;)
    {
        if((conn = accept(socket, (sockaddr *) &addr, &sizeof_addr)) == -1)
        {
            perror("accept");
        }
        else
        {
            puts("server: accept");

            if(!fork()) // actually not necessary, only got 1 client
            {
                close(socket);
                // char *buf = new char[1024*1024];
                // read(conn, buf, 1024*1024); // black hole
                // server never reads
                break;
            }
        }
    }
}

void do_send(int socket, const char* buf, unsigned int size/*,  bool nonblock = false */)
{
    unsigned int sent = 0;
    unsigned int count = 0;

    while(sent < size)
    {
        int n = send(socket, &buf[sent], size - sent, 0);
        // int n = send(socket, &buf[sent], size - sent, MSG_DONTWAIT);

        if (n == -1)
        {
            if(errno == EAGAIN)
            {
                printf(".");
                printf("\n");
            }
            else
            {
                perror("\nsend");
                return;
            }
        }
        else
        {
            sent += n;
            printf("  --> sent a chunk of %u bytes (send no. %u, total sent = %u)\n", n, ++count, sent);
        }
    }
}

void client()
{
    const unsigned int max_size = 64*1024*1024; // sending up to 64MiB in one call

    sleep(1); // give server a second to start up

    int socket = create_socket();


    unsigned int send_buffer_size = 0;
    unsigned int len = sizeof(send_buffer_size);

    if(getsockopt(socket, SOL_SOCKET, SO_SNDBUF, &send_buffer_size, &len))
        perror("getsockopt");

    // Linux internally doubles the buffer size, and getsockopt reports the doubled size
    printf("send buffer size = %u  (doubled, actually %u)\n", send_buffer_size, send_buffer_size/2);

    if(socket == -1)
    {
        puts("no good");
        exit(1);
    }

    char *buf = new char[max_size]; // uninitialized contents, but who cares

    for(unsigned int size = 65536; size <= max_size; size += 16384)
    {
        printf("attempting to send %u bytes\n", size);
        do_send(socket, buf, size);
    }
    puts("all done");
    delete buf;
}


int main()
{
    setvbuf(stdout, NULL, _IONBF, 0);
    if(fork() > 0) server(); else client();
    return 0;
}

compile and run

g++ -g -Wall -o send-blocking-and-server-never-read code-of-damon.cpp
./send-blocking-and-server-never-read > log1.log 2>&1

log1.log content

server: listening
client: connectedserver: accept

send buffer size = 2626560  (doubled, actually 1313280)
attempting to send 65536 bytes
  --> sent a chunk of 65536 bytes (send no. 1, total sent = 65536)
attempting to send 81920 bytes
  --> sent a chunk of 81920 bytes (send no. 1, total sent = 81920)
attempting to send 98304 bytes
  --> sent a chunk of 98304 bytes (send no. 1, total sent = 98304)
attempting to send 114688 bytes
  --> sent a chunk of 114688 bytes (send no. 1, total sent = 114688)
attempting to send 131072 bytes
  --> sent a chunk of 131072 bytes (send no. 1, total sent = 131072)
attempting to send 147456 bytes
  --> sent a chunk of 147456 bytes (send no. 1, total sent = 147456)
attempting to send 163840 bytes
  --> sent a chunk of 163840 bytes (send no. 1, total sent = 163840)
attempting to send 180224 bytes
  --> sent a chunk of 180224 bytes (send no. 1, total sent = 180224)
attempting to send 196608 bytes
  --> sent a chunk of 196608 bytes (send no. 1, total sent = 196608)
attempting to send 212992 bytes
  --> sent a chunk of 212992 bytes (send no. 1, total sent = 212992)
attempting to send 229376 bytes
  --> sent a chunk of 229376 bytes (send no. 1, total sent = 229376)
attempting to send 245760 bytes
  --> sent a chunk of 245760 bytes (send no. 1, total sent = 245760)
attempting to send 262144 bytes
  --> sent a chunk of 262144 bytes (send no. 1, total sent = 262144)
attempting to send 278528 bytes
  --> sent a chunk of 278528 bytes (send no. 1, total sent = 278528)
attempting to send 294912 bytes

then comment int n = send(socket, &buf[sent], size - sent, 0); and uncomment int n = send(socket, &buf[sent], size - sent, MSG_DONTWAIT);

compile and run again

g++ -g -Wall -o send-nonblocking-and-server-never-read code-of-damon.cpp
./send-nonblocking-and-server-never-read > log2.log 2>&1

log2.log content

server: listening
server: accept
client: connected
send buffer size = 2626560  (doubled, actually 1313280)
attempting to send 65536 bytes
  --> sent a chunk of 65536 bytes (send no. 1, total sent = 65536)
attempting to send 81920 bytes
  --> sent a chunk of 81920 bytes (send no. 1, total sent = 81920)
attempting to send 98304 bytes
  --> sent a chunk of 98304 bytes (send no. 1, total sent = 98304)
attempting to send 114688 bytes
  --> sent a chunk of 114688 bytes (send no. 1, total sent = 114688)
attempting to send 131072 bytes
  --> sent a chunk of 131072 bytes (send no. 1, total sent = 131072)
attempting to send 147456 bytes
  --> sent a chunk of 147456 bytes (send no. 1, total sent = 147456)
attempting to send 163840 bytes
  --> sent a chunk of 163840 bytes (send no. 1, total sent = 163840)
attempting to send 180224 bytes
  --> sent a chunk of 180224 bytes (send no. 1, total sent = 180224)
attempting to send 196608 bytes
  --> sent a chunk of 196608 bytes (send no. 1, total sent = 196608)
attempting to send 212992 bytes
  --> sent a chunk of 212992 bytes (send no. 1, total sent = 212992)
attempting to send 229376 bytes
  --> sent a chunk of 229376 bytes (send no. 1, total sent = 229376)
attempting to send 245760 bytes
  --> sent a chunk of 245760 bytes (send no. 1, total sent = 245760)
attempting to send 262144 bytes
  --> sent a chunk of 262144 bytes (send no. 1, total sent = 262144)
attempting to send 278528 bytes
  --> sent a chunk of 278528 bytes (send no. 1, total sent = 278528)
attempting to send 294912 bytes
  --> sent a chunk of 178145 bytes (send no. 1, total sent = 178145)
.
.
.
.
.
.
// endless .

Compare the last output of log1.log and log2.log and you can tell that a blocking send blocks when there's no enough buffer to fit all 294912 bytes while a non-blocking send performs a partial write. This conforms to conclusion #1.

Special thanks to @user207421's different opinion that leads me on more searching.

Answer 2

Q: Does "send()" always return the whole buffer?

A: No, not necessarily.

From Beej's Guide: * http://beej.us/guide/bgnet/html/multi/syscalls.html#sendrecv

send() returns the number of bytes actually sent out—this might be less than the number you told it to send! See, sometimes you tell it to send a whole gob of data and it just can't handle it. It'll fire off as much of the data as it can, and trust you to send the rest later. Remember, if the value returned by send() doesn't match the value in len, it's up to you to send the rest of the string. The good news is this: if the packet is small (less than 1K or so) it will probably manage to send the whole thing all in one go. Again, -1 is returned on error, and errno is set to the error number.

Q: Does "recv()" always read the whole buffer?

A: No, absolutely not. You should never assume the buffer you've received is "the whole message". Or assume the message you receive is from one, single message.

Here's a good, short explanation. It's for Microsoft/C#, but it's applicable to all sockets I/O, in any language:

• http://blogs.msdn.com/b/joncole/archive/2006/03/20/simple-message-framing-sample-for-tcp-socket.aspx

Answer 3

Nope, it doesn't.

For reference, see the man page for send:

When the message does not fit into the send buffer of the socket, send() normally blocks, unless the socket has been placed in nonblocking I/O mode. In nonblocking mode it would fail with the error EAGAIN or EWOULDBLOCK in this case. The select(2) call may be used to determine when it is possible to send more data.

Answer 4

The answer is in another section of man 2 send:

   When the message does not fit into  the  send  buffer  of  the  socket,
   send()  normally blocks, unless the socket has been placed in nonblock‐
   ing I/O mode.  In nonblocking mode it would fail with the error  EAGAIN
   or  EWOULDBLOCK in this case.  The select(2) call may be used to deter‐
   mine when it is possible to send more data.

Or, alternatively, the POSIX version (man 3p send):

   If  space is not available at the sending socket to hold the message to
   be transmitted, and the socket file descriptor does not have O_NONBLOCK
   set,  send()  shall  block  until  space is available.  If space is not
   available at the sending socket to hold the message to be  transmitted,
   and  the  socket file descriptor does have O_NONBLOCK set, send() shall
   fail. The select() and poll() functions can be used to  determine  when
   it is possible to send more data.

So, while a read of partial data is common, a partial send in blocking mode should not happen (barring implementation details).