Standalone ASIO library has different behaviour on OSX and Linux systems after write error

Question

I have noticed that on OSX asio::async_write function always calls handler callback. But on linux (Ubuntu 18.04) after async_write operation completes with error 3 times (Connection reset by peer or Broken pipe) handler callback is not called anymore after next call to async_write.

Please take a look at the code example:

    asio::io_service ioService;

    asio::ip::tcp::resolver resolver(ioService);

    // ---- Initialize server -----
    auto acceptor = make_unique<asio::ip::tcp::acceptor>(ioService,
        resolver.resolve(asio::ip::tcp::resolver::query(asio::ip::tcp::v4(), "localhost", "12345"))->endpoint());;
    asio::ip::tcp::socket serverSocket(ioService);
    std::promise<void> connectedPromise;
    std::promise<void> disconnectedPromise;
    std::vector<uint8_t> readBuffer(1);
    acceptor->async_accept(serverSocket, [&](asio::error_code errorCode) {
        std::cout << "Socket accepted!" << std::endl;
        connectedPromise.set_value();
        serverSocket.async_read_some(asio::buffer(readBuffer), [&](asio::error_code errorCode, std::size_t length) {
            if (errorCode) {
                std::cout << "Read error: " << errorCode.message() << std::endl;
                disconnectedPromise.set_value();
            }
        });
    });

    // ----- Initialize client --------
    asio::ip::tcp::socket clientSocket(ioService);
    asio::connect(clientSocket, resolver.resolve({asio::ip::tcp::v4(), "localhost", "12345"}));

    // ----- Start io service loop
    std::thread mainLoop([&]() {
        ioService.run();
    });

    connectedPromise.get_future().get(); // Wait until connected

    // ----- Perform 10 async_write operations with 100 ms delay --------

    std::promise<void> done;
    std::atomic<int> writesCount{0};
    std::vector<uint8_t> writeBuffer(1);

    std::function<void (const asio::error_code&, std::size_t)> writeHandler = [&](const asio::error_code& errorCode, std::size_t) -> void {
        if (errorCode) {
            std::cout << errorCode.message() << std::endl;
        }
        if (++writesCount < 10) {
            std::this_thread::sleep_for(std::chrono::milliseconds(100));
            asio::async_write(serverSocket, asio::buffer(writeBuffer), writeHandler);
        } else {
            done.set_value();
        }
    };

    asio::async_write(serverSocket, asio::buffer(writeBuffer), writeHandler);

    clientSocket.close(); // Perform disconnect from client side
    disconnectedPromise.get_future().get(); // Wait until disconnected

    std::cout << "Waiting for all operations complete" << std::endl;
    done.get_future().get(); // Wait until all 10 async_write operations complete
    std::cout << "All operations complete" << std::endl;

    ioService.stop();
    mainLoop.join();

Output on OSX:

Socket accepted!
Broken pipe
Read error: Connection reset by peer
Broken pipe
Waiting for all operations complete
Broken pipe
Broken pipe
Broken pipe
Broken pipe
Broken pipe
Broken pipe
Broken pipe
All operations complete

Output on Ubuntu 18.04:

Socket accepted!
Read error: End of file
Connection reset by peer
Waiting for all operations complete
Broken pipe
Broken pipe

Linux version hangs on done.get_future().get() line because async_write completion handler is not called after several Broken pipe errors. I expect that any async_write operation should lead to handler call regardless of the socket status as in OSX version. Is it a bug in linux version?

Asio version: 1.14.0 (standalone)

Answer 1

You have condition race on ioService.run().

The reference states:

The run() function blocks until all work has finished and there are no more handlers to be dispatched.

You have to call reset() on ioService if service stopped working due to lack of handlers.

A normal exit from the run() function implies that the io_context object is stopped (the stopped() function returns true). Subsequent calls to run(), run_one(), poll() or poll_one() will return immediately unless there is a prior call to restart().

The diagram below shows where condition race occurs:

main thread                                   background thread

[1] async_accept

[2] ioService.run()
                                              [3] handler for async_accept is called 
                                                  connectedPromise.set_value();
                                              [4] async_read_some 

[5] connectedPromise.get_future().get();

---> now here is a problem 
                   What [6.a] or [6.b] will be called as first?

[6.a] async_write which can push 
      a new handler to be processed      
                                       or 
                                          [6.b] handler for async_read_some 
                                               if this handler was called,
                                               ioService::run() ends, and you have to call reset on 
                                               it to accept new incoming handlers

(In square brackets are all steps in time order)

In your case 6.b happens. Handler for async_read_some is called as first, and in this handler you don't initiate any new handlers. As a result, ioService::run() stops, and handler for async_write will not be invoked.

Try to use executor_work_guard to prevent ioService::run() from stopping when there is no handlers to be dispatched.