Segmentation Fault on Linux C++ Boost Asio Server build

Question

Segmentation Fault in Boost Asio Server on Linux during flat_buffer::prepare() call

Problem Description:

I have a C++20 server application using Boost.Asio 1.85 that works perfectly on Windows but crashes on Ubuntu Linux with a segmentation fault. The crash occurs when handling an HTTP request, specifically inside Boost.Beast's flat_buffer::prepare() function. I suspect it has something to do with memory allocation or access, possibly related to NUMA/CPU affinity optimizations or thread handling on Linux.

#include <atomic>
#include <boost/asio.hpp>
#include <boost/beast.hpp>
#include <cassert>
#include <concurrentqueue.h>
#include <condition_variable>
#include <numa.h>
namespace asio  = boost::asio;
namespace beast = boost::beast;
namespace http  = beast::http;
using tcp       = asio::ip::tcp;
static constexpr uint16_t SERVERPORT = 7878;

class JsonEchoServer final : public std::enable_shared_from_this<JsonEchoServer> {
  public:
    explicit JsonEchoServer(asio::io_context& io_context, asio::ip::port_type port)
        : io_context_(io_context)
        , acceptor_(io_context, {tcp::v4(), port})
        , thread_pool_(std::max(1u, std::thread::hardware_concurrency())) {
        assert(acceptor_.is_open() && "Acceptor must be open.");
    }

    JsonEchoServer() = delete;

    void run_server() {
        co_spawn(io_context_, listener(), asio::detached);

        std::size_t              num_threads = std::max<std::size_t>(1, std::thread::hardware_concurrency());
        std::vector<std::thread> threads;
        threads.reserve(num_threads);

        for (std::size_t i = 0; i < num_threads; ++i) {
            threads.emplace_back([this]() {
                optimize_for_platform();
                io_context_.run();
            });
        }

        for (auto& th : threads) {
            if (th.joinable())
                th.join();
        }
    }

  private:
    asio::io_context& io_context_;
    tcp::acceptor     acceptor_;

    class ThreadPool {
      public:
        explicit ThreadPool(std::size_t num_threads) : stop_(false) {
            workers_.reserve(num_threads);
            for (std::size_t i = 0; i < num_threads; ++i) {
                workers_.emplace_back([this]() {
                    optimize_for_platform();
                    worker_thread();
                });
            }
        }

        ~ThreadPool() {
            stop_.store(true, std::memory_order_release);
            condition_.notify_all();
            for (auto& worker : workers_) {
                if (worker.joinable())
                    worker.join();
            }
        }

        void enqueue(std::function<void()> task) {
            tasks_.enqueue(std::move(task));
            condition_.notify_one();
        }

      private:
        moodycamel::ConcurrentQueue<std::function<void()>> tasks_;
        std::vector<std::thread>                           workers_;
        std::atomic<bool>                                  stop_;
        std::condition_variable                            condition_;
        std::mutex                                         mutex_;

        void worker_thread() {
            optimize_for_platform();
            while (!stop_.load(std::memory_order_acquire)) {
                std::function<void()> task;
                if (tasks_.try_dequeue(task)) {
                    task();
                } else {
                    std::unique_lock<std::mutex> lock(mutex_);
                    condition_.wait_for(lock, std::chrono::milliseconds(1));
                }
            }
        }

        static void optimize_for_platform() {
#ifdef linux
            cpu_set_t cpuset;
            CPU_ZERO(&cpuset);
            int cpu = sched_getcpu();
            assert(cpu >= 0 && "sched_getcpu() failed on Linux.");
            CPU_SET(cpu, &cpuset);
            int set_res = pthread_setaffinity_np(pthread_self(), sizeof(cpu_set_t), &cpuset);
            assert(set_res == 0 && "pthread_setaffinity_np() failed.");
            int node = numa_node_of_cpu(cpu);
            numa_run_on_node(node);
            numa_set_preferred(node);
#endif
        }
    };

    ThreadPool thread_pool_;

    asio::awaitable<void> handle_resultaten_endpoint(                     //
        [[maybe_unused]] tcp::socket                              socket, //
        [[maybe_unused]] http::request<http::dynamic_body> const& req) {
        // ... implementation ...
        co_return;
    }

    asio::awaitable<void> handle_bestand_other_generator_endpoint(        //
        [[maybe_unused]] tcp::socket                              socket, //
        [[maybe_unused]] http::request<http::dynamic_body> const& req) {
        // ... implementation ...
        co_return;
    }

    asio::awaitable<void> handle_request(tcp::socket socket) {
        socket.set_option(tcp::no_delay(true));
        beast::flat_buffer                       buffer;
        http::request_parser<http::dynamic_body> req_parser;
        req_parser.body_limit(std::numeric_limits<std::uint64_t>::max());

        co_await http::async_read(socket, buffer, req_parser, asio::deferred);
        http::request<http::dynamic_body> req(std::move(req_parser.get()));

        static constexpr std::string_view resultaten_sv    = "/resultaten";
        static constexpr std::string_view bestand_other_sv = "/bestand_other";
        std::string_view                  target_sv        = req.target();

        if (target_sv.starts_with(bestand_other_sv)) {
            co_await handle_bestand_other_generator_endpoint(std::move(socket), std::move(req));
        } else if (target_sv.starts_with(resultaten_sv)) {
            co_await handle_resultaten_endpoint(std::move(socket), std::move(req));
        } else {
            http::response<http::string_body> res;
            res.version(req.version());
            res.result(http::status::not_found);
            res.set(http::field::content_type, "application/json");
            res.body() = R"({"Status":"Fataal","Omschrijving":"Endpoint niet gevonden"})";
            res.prepare_payload();
            co_await http::async_write(socket, res, asio::deferred);
            boost::beast::error_code shutdown_ec;
            socket.shutdown(tcp::socket::shutdown_send, shutdown_ec);
        }
        co_return;
    }

    template <typename Func> auto offload_to_threadpool(Func&& func) -> asio::awaitable<decltype(func())> {
        using result_type = decltype(func());
        auto token        = asio::use_awaitable;
        co_return co_await asio::async_initiate<asio::use_awaitable_t<>,
                                                void(boost::system::error_code, result_type)>(
            [this, f = std::forward<Func>(func)](auto handler) mutable {
                using handler_type = std::decay_t<decltype(handler)>;
                auto handler_ptr   = std::make_shared<handler_type>(std::move(handler));
                thread_pool_.enqueue([f = std::move(f), handler_ptr, this]() mutable {
                    try {
                        result_type result = f();
                        asio::post(io_context_, [handler_ptr, r = std::move(result)]() mutable {
                            (*handler_ptr)(boost::system::error_code(), std::move(r));
                        });
                    } catch (...) {
                        assert(false && "Caught unexpected exception in offload_to_threadpool.");
                        asio::post(io_context_, [handler_ptr]() {
                            (*handler_ptr)(asio::error::operation_aborted, result_type{});
                        });
                    }
                });
            },
            token);
    }

    asio::awaitable<void> listener() {
        while (true) {
            tcp::socket socket = co_await acceptor_.async_accept(asio::use_awaitable);
            auto        self   = shared_from_this();
            asio::co_spawn(
                io_context_,
                [this, self, s = std::move(socket)]() mutable { return handle_request(std::move(s)); },
                asio::detached);
        }
    }

    static void optimize_for_platform() {
#ifdef linux
        cpu_set_t cpuset;
        CPU_ZERO(&cpuset);
        int cpu = sched_getcpu();
        assert(cpu >= 0 && "sched_getcpu() failed on Linux.");
        CPU_SET(cpu, &cpuset);
        int set_res = pthread_setaffinity_np(pthread_self(), sizeof(cpu_set_t), &cpuset);
        assert(set_res == 0 && "pthread_setaffinity_np() failed.");
        int node = numa_node_of_cpu(cpu);
        numa_run_on_node(node);
        numa_set_preferred(node);
#endif
    }
};

inline static void launch_server_process() {
    asio::io_context io_context;
    auto             server = std::make_shared<JsonEchoServer>(io_context, SERVERPORT);
    server->run_server();
}

int main() {
    launch_server_process();
    return 0;
}

Issue Details:

The server runs fine on Windows but causes a segmentation fault on Linux when a request arrives.

The crash occurs within Boost.Beast's flat_buffer::prepare() call during asynchronous reading of the HTTP request.

This suggests possible memory corruption or misuse of flat_buffer—perhaps due to improper thread usage or platform-specific configuration on Linux.

Question:

What exactly is causing the segmentation fault in flat_buffer::prepare() on Linux, and how can I resolve it?

I have gone through the code extensively, examining it for possible memory corruption and thread safety issues around the use of flat_buffer. I checked that there were no concurrent accesses without proper synchronization and that all buffer objects were managed correctly.

I also investigated the platform-specific optimizations, such as CPU affinity and NUMA calls within optimize_for_platform(), considering that these Linux-specific calls might cause unexpected behavior or crashes if, for example, required NUMA libraries were not linked or the hardware did not support certain features.

To gain more insight into the segmentation fault, I used debugging tools like GDB, Valgrind, and AddressSanitizer. I attempted to get detailed information from these tools, but despite these efforts, the stack traces and error messages didn’t point me directly to the root cause of the crash inside flat_buffer::prepare(). This is the bt output when running gdb : https://drive.google.com/file/d/1MjXw_nunRuu8L3-rIirmGUPny_tpRDmK/view?usp=sharing

Thread 23 "ServerExecutable" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff83f6d6c0 (LWP 9305)]
0x000055555587ea05 in boost::beast::basic_flat_buffer<allocator<?> >::prepare (this=0x7fff70001298, n=512) at ISA_Software/ServerExecutable/../Kernel/boost/beast/core/impl/flat_buffer.hpp:322
322     auto const len = size();

#0  0x000055555587ea05 in boost::beast::basic_flat_buffer<allocator<?> >::prepare (this=0x7fff70001298, n=512) at ISA_Software/ServerExecutable/../Kernel/boost/beast/core/impl/flat_buffer.hpp:322
#1  0x000055555587ca1f in boost::beast::detail::dynamic_buffer_prepare<boost::beast::basic_flat_buffer<?>, boost::beast::http::error> (buffer=..., size=512, ec=..., ev=boost::beast::http::error::buffer_overflow) at ISA_Software/ServerExecutable/../Kernel/boost/beast/core/detail/buffer.hpp:64
#2  0x000055555587b10e in boost::beast::http::detail::read_some_op<boost::asio::basic_stream_socket<?>, boost::beast::basic_flat_buffer<?>, true>::operator()<detail::composed_op<?> >(detail::composed_op<boost::beast::http::detail::read_some_op<?>, detail::composed_work<?>, detail::composed_op<?>, void (error_code, unsigned long)>&, error_code, unsigned long) (this=0x7fff82801408, self=..., ec=..., bytes_transferred=0) at ISA_Software/ServerExecutable/../Kernel/boost/beast/http/impl/read.hpp:206
...
#8  0x0000555555869d85 in boost::beast::http::async_read_some<boost::asio::basic_stream_socket<?>, boost::beast::basic_flat_buffer<?>, true, detail::composed_op<?> >(boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::any_io_executor>&, boost::beast::basic_flat_buffer<allocator<?> >&, boost::beast::http::basic_parser<true>&, detail::composed_op<boost::beast::http::detail::read_op<?>, detail::composed_work<?>, detail::awaitable_handler<?>, void (error_code, unsigned long)>&&) (stream=..., buffer=..., parser=..., handler=...) at ISA_Software/ServerExecutable/../Kernel/boost/beast/http/impl/read.hpp:463
...
#16 0x0000555555863759 in detail::awaitable_frame_base<boost::asio::any_io_executor>::resume (this=0x7fff70001660) at ISA_Software/ServerExecutable/../Kernel/boost/asio/impl/awaitable.hpp:503
...
#39 0x00005555557b352a in boost::asio::co_spawn<boost::asio::io_context, JsonEchoServer::listener()::{lambda()#1}, boost::asio::detached_t const&>(boost::asio::io_context&, JsonEchoServer::listener()::{lambda()#1}&&, boost::asio::detached_t const&, boost::asio::constraint<is_convertible<?>::value, int>::type) (ctx=..., f=..., token=...) at ISA_Software/ServerExecutable/../Kernel/boost/asio/impl/co_spawn.hpp:442
#40 0x000055555575168b in JsonEchoServer::listener(_ZN14JsonEchoServer8listenerEv.Frame *) (frame_ptr=0x55555609cb00) at ISA_Software/ServerExecutable/CoreSuite/Include/JsonEchoServer.hpp:469
#41 0x00005555557656f7 in __n4861::coroutine_handle<void>::resume (this=0x55555609cb10) at /usr/include/c++/14/coroutine:137
...
#51 0x0000555555765351 in detail::scheduler::do_run_one (this=0x55555609d210, lock=..., this_thread=..., ec=...) at ISA_Software/ServerExecutable/../Kernel/boost/asio/detail/impl/scheduler.ipp:493
#52 0x0000555555764cb9 in detail::scheduler::run (this=0x55555609d210, ec=...) at ISA_Software/ServerExecutable/../Kernel/boost/asio/detail/impl/scheduler.ipp:210
#53 0x00005555557659e3 in boost::asio::io_context::run (this=0x7fffffffdb50) at ISA_Software/ServerExecutable/../Kernel/boost/asio/impl/io_context.ipp:64
#54 0x00005555557b122e in JsonEchoServer::run_server()::{lambda()#1}::operator()() const (__closure=0x5555560a82d8) at ISA_Software/ServerExecutable/CoreSuite/Include/JsonEchoServer.hpp:52
...
#61 0x00007ffff789ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447

I increased logging and added assertions around the call to prepare() and in the functions that led up to it. By inspecting the internal state of the buffer and the values of parameters passed in, I hoped to catch any anomalies, but these checks did not reveal the source of the fault.

Additionally, I reviewed the implementation of my thread pool and the use of shared_from_this() to ensure that objects and resources were managed properly without lifetime issues. Even though this confirmed correct usage in those areas, it didn’t resolve the segmentation fault.

Finally, I tried adjusting the NUMA settings by temporarily disabling or using less aggressive settings, hoping that this would prevent the crash. Unfortunately, the segmentation fault persisted, indicating that the issue was not straightforwardly solved by changing these platform-specific optimizations.