What is the condition for completion handler of basic_stream_socket::async_write_some to be called?

Question

The documentation for basic_stream_socket::async_write_some states, that the completion handler is called "when the write operation completes". But what exactly does that mean? I can think at least of two things

• when the kernel (networking stack in particular) takes responsibility for the data
• when the networking stack on the remote end has acknowledged (as in TCP ACK) the data, that was sent and the acknowledgement has reached the local networking stack

Answer 1

Tried to come up with a test program:

Coliru

#include <boost/asio.hpp>
#include <iostream>

using namespace boost::asio;
using ip::tcp;
using boost::system::error_code;

using namespace std::chrono_literals;
auto now       = &std::chrono::high_resolution_clock::now;
auto sleep_for = [](auto dur) { std::this_thread::sleep_for(dur); };
auto timestamp = [start = now()] { return (now() - start)/1.0ms; };

int main() {
    static constexpr size_t msglen = 16 << 20; // 16 mib
    thread_pool io(1);
    tcp::acceptor a(io, {{}, 7878});
    a.set_option(tcp::acceptor::reuse_address(true));
    a.listen();

#define CHECKED_OPTION(s, name, requested) do { \
    tcp::socket::name option(requested); \
    /*s.set_option(option);*/ \
    s.get_option(option); \
    std::cout << " " << #name << ":" << option.value(); \
} while (0)

    a.async_accept([=](error_code ec, tcp::socket&& con) {
        std::cout << timestamp() << "ms accept " << ec.message();
        std::cout << " " << con.remote_endpoint(ec);

        con.set_option(tcp::no_delay(true));
        CHECKED_OPTION(con, receive_buffer_size, 100);
        CHECKED_OPTION(con, send_buffer_size, 100);

        std::cout << std::endl;

        if (!ec) {
            sleep_for(1s);
            std::cout << timestamp() << "ms start write" << std::endl;
            auto xfr = con.write_some(buffer(std::string(msglen, '*')), ec);
            std::cout << timestamp() << "ms write completed: " << xfr << "/" << msglen << " (" << ec.message() << ")" << std::endl;
        }
    });

    {
        tcp::socket s(io);
        sleep_for(1s);
        std::cout << timestamp() << "ms connecting" << std::endl;
        s.connect({{}, 7878});
        std::cout << timestamp() << "ms connected";
        CHECKED_OPTION(s, receive_buffer_size, 100);
        CHECKED_OPTION(s, send_buffer_size, 100);
        std::cout << std::endl;

        sleep_for(3s);
        std::cout << timestamp() << "ms disconnecting" << std::endl;
    }
    std::cout << timestamp() << "ms disconnected" << std::endl;

    a.cancel();
    io.join();
}

Note how we make sure to send more data than is being read by a wide margin to saturate any buffering involved. (We actually donot read any data from the client socket at all)

It prints (on Coliru):

1000.48ms connecting
1001.07ms connected receive_buffer_size:530904 send_buffer_size:1313280
1001.23ms accept Success 127.0.0.1:41614 receive_buffer_size:531000 send_buffer_size:1313280
2001.64ms start write
4001.37ms disconnecting
4001.62ms disconnected
4013.33ms write completed: 4481610/16777216 (Success)

It is clear that

• write_some is only complete when the packages are ACK-ed, and the actual number of bytes transferred is returned.
• the kernel ACKs packages independently of the application layer

In fact the packages may arrive out of order, in which case the kernel ACKs them individually before the sequencing for the application to read the data via the socket API.

Buffering

Buffering is inevitable, but can be tuned within limits. E.g. uncommenting this line from the CHECKED_OPTION macro:

s.set_option(option); \

Gives different output (Live On Coliru):

1000.44ms connecting
1001.08ms connected receive_buffer_size:1152 send_buffer_size:2304
1001.31ms accept Success 127.0.0.1:41618 receive_buffer_size:1152 send_buffer_size:2304
2001.88ms start write
4001.42ms disconnecting
4001.61ms disconnected
4008.21ms write completed: 43776/16777216 (Success)