Memory Leak at async_handshake using boost beast and openssl

Question

I'm using OpenSSL 1.1.1b and Boost 1.68 to create a simple server using https.

I followed the examples provided by boost beast and in particular the advance server flex.

The application seems to work properly. I can accept https session and also wss sessions.

The problem is when I exit from the application where the Visual Leak Detector finds 16 memory leaks that target at:

c:\openssl-1.1.1b\crypto\mem.c (233): abc.exe!CRYPTO_zalloc
    c:\openssl-1.1.1b\crypto\err\err.c (716): abc.exe!ERR_get_state + 0x17 bytes
    c:\openssl-1.1.1b\crypto\err\err.c (443): abc.exe!ERR_clear_error + 0x5 bytes
    c:\usr\work\abc_repo\ext\boost_1_68_0\boost\asio\ssl\detail\impl\engine.ipp (235): abc.exe!boost::asio::ssl::detail::engine::perform
    c:\usr\work\abc_repo\ext\boost_1_68_0\boost\asio\ssl\detail\impl\engine.ipp (137): abc.exe!boost::asio::ssl::detail::engine::handshake

I modified the pattern of the http session from the original boost beast code but it should perform exactly the same things.

I've tried to understand if the memory leaks increase based on the number of connections but it seems not. I don't understand how to get rid of this problem.

Following the code I used. First a based http session class

class CApplicationServerBaseHttpSession
{

public:
    std::shared_ptr<CApplicationServerSharedState> m_state = nullptr;
    CApplicationServerHttpQueue m_queue;

    // The parser is stored in an optional container so we can
    // construct it from scratch it at the beginning of each new message.
    boost::optional<boost::beast::http::request_parser<boost::beast::http::string_body>> parser_;

protected:
    boost::asio::steady_timer m_timer;
    boost::beast::flat_buffer buffer_;
    boost::log::sources::severity_channel_logger<boost::log::trivial::severity_level> m_Logger{boost::log::keywords::channel = LOG_APPLICATION_SERVER_CHANNEL_ID};
    boost::asio::strand<boost::asio::io_context::executor_type> m_strand;

public:
    // Construct the session
    CApplicationServerBaseHttpSession(
        boost::asio::io_context& ioc,
        boost::beast::flat_buffer buffer,
        std::shared_ptr<CApplicationServerSharedState> const& state)
        : m_state(state)
        , m_strand(ioc.get_executor())
        , m_timer(ioc,
            (std::chrono::steady_clock::time_point::max)()
        )
        , m_queue(*this)
        , buffer_(std::move(buffer))
    {
    }

    void DoRead();

    void OnRead(boost::system::error_code ec);

    void OnWrite(boost::system::error_code ec, bool close);

    virtual void WriteRequestStringBody(boost::beast::http::response<boost::beast::http::string_body> & msg) = 0;
    virtual void WriteRequestFileBody(boost::beast::http::response<boost::beast::http::file_body> & msg) = 0;

protected:

    virtual void ReadRequest() = 0;


    virtual void DoEof() = 0;

    virtual std::string GetRemoteAddress() = 0;

    virtual void MakeWebSocketSession(boost::beast::http::request<boost::beast::http::string_body> req) = 0;
};

Here the implementation:

void CApplicationServerBaseHttpSession::DoRead()
{
    // Set the timer
    m_timer.expires_after(std::chrono::seconds(OCV_HTTP_SESSION_TIMER_EXPIRE_AFTER));

    // Construct a new parser for each message
    parser_.emplace();

    // Apply a reasonable limit to the allowed size
    // of the body in bytes to prevent abuse.
    parser_->body_limit(HTTP_BODY_LIMIT);

    this->ReadRequest();

}

void CApplicationServerBaseHttpSession::OnRead(boost::system::error_code ec)
{
// Happens when the timer closes the socket
    if(ec == boost::asio::error::operation_aborted)
        return;

    // This means they closed the connection
    if(ec == http::error::end_of_stream)
        return this->DoEof();

    if(ec == boost::asio::ssl::error::stream_truncated){
        // "stream truncated" means that the other end closed the connection abruptly.
        return warning(ec, "Http read", m_Logger);
    }

    if(ec)
        return fail(ec, "Http read", m_Logger);

    // See if it is a WebSocket Upgrade
    if(websocket::is_upgrade(parser_->get())) {

        // Get a websocket request handler to execute operation as authentication and authorization
        // If these steps are allowed than the websocket session will be started
        std::shared_ptr<CApplicationServerWsApiBase> endpointWs = m_state->GetEndpointWs(parser_->get().target().to_string());

        if(endpointWs) {
            int endpointErrorDefault = endpointWs->HandleRequest(parser_->get());
            if(endpointErrorDefault > 0) { // Success Auth

                // Make timer expire immediately, by setting expiry to time_point::min we can detect
                // the upgrade to websocket in the timer handler
                m_timer.expires_at((std::chrono::steady_clock::time_point::min)());

                // Transfer the stream to a new WebSocket session
                return MakeWebSocketSession(parser_->release());

            } else {
                // Authentication or Authorization failed
                m_queue(endpointWs->GetResponseError(parser_->get(), endpointErrorDefault));
                return;
            }
        } else {
            // Wrong endpoint called: BadRequest
            std::shared_ptr<CApplicationServerApiBase> endpoint = m_state->GetEndpoint(ApiURI::REQUEST_NOT_IMPLEMENTED);
            if(endpoint) {
                endpoint->HandleRequest(m_state->GetDocRoot(), parser_->release(), m_queue);
            }
            return;
        }

    }

    BOOST_LOG_SEV(m_Logger, boost::log::trivial::trace) <<
        "Request From: " <<
        this->GetRemoteAddress() <<
        " Request Target: " <<
        parser_->get().target().to_string();

    std::shared_ptr<CApplicationServerApiBase> endpoint = m_state->GetEndpoint(parser_->get().target().to_string());

    if(endpoint) {
        endpoint->HandleRequest(m_state->GetDocRoot(), parser_->release(), m_queue);
    }


    // If we aren't at the queue limit, try to pipeline another request
    if(!m_queue.IsFull()) {
        DoRead();
    }
}

void  CApplicationServerBaseHttpSession::OnWrite(boost::system::error_code ec, bool close)
{
// Happens when the timer closes the socket
    if(ec == boost::asio::error::operation_aborted)
        return;

    if(ec)
        return fail(ec, "write", m_Logger);

    if(close) {
        // This means we should close the connection, usually because
        // the response indicated the "Connection: close" semantic.
        return this->DoEof();
    }

    // Inform the queue that a write completed
    if(m_queue.OnWrite()) {
        // Read another request
        DoRead();
    }
}

The https session:

class COcvApplicationServerHttpSessionSSL 
    : public std::enable_shared_from_this<COcvApplicationServerHttpSessionSSL>
    , public CApplicationServerBaseHttpSession
{

public:


public:
    COcvApplicationServerHttpSessionSSL(boost::asio::ip::tcp::socket&& socket,boost::asio::ssl::context& ctx, boost::beast::flat_buffer&& buffer, std::shared_ptr<CApplicationServerSharedState> const& state);
    ~COcvApplicationServerHttpSessionSSL();

    // Called by the base class
    boost::beast::ssl_stream<boost::asio::ip::tcp::socket>& Stream();
    boost::beast::ssl_stream<boost::asio::ip::tcp::socket> ReleaseStream();
    void DoTimeout();


    // Start the asynchronous operation
    void Run();

    void OnHandshake(boost::system::error_code ec, std::size_t bytes_used);

    void OnShutdown(boost::system::error_code ec);

    void OnTimer(boost::system::error_code ec);

private:

public:
    boost::beast::ssl_stream<boost::asio::ip::tcp::socket> m_stream;
    bool m_eof = false;

protected:

    // Inherited via COcvApplicationServerBaseHttpSession
    virtual void ReadRequest() override;

    virtual void WriteRequestStringBody(boost::beast::http::response<boost::beast::http::string_body> & msg) override;
    virtual void WriteRequestFileBody(boost::beast::http::response<boost::beast::http::file_body> & msg) override;

    virtual void DoEof() override;

    virtual std::string GetRemoteAddress() override;

    virtual void MakeWebSocketSession(boost::beast::http::request<boost::beast::http::string_body> req) override;

};

and at the end the implementatition

COcvApplicationServerHttpSessionSSL::COcvApplicationServerHttpSessionSSL(tcp::socket&& socket, ssl::context & ctx, beast::flat_buffer&& buffer, std::shared_ptr<CApplicationServerSharedState> const & state)
    : CApplicationServerBaseHttpSession(
        socket.get_executor().context(),
        std::move(buffer),
        state)
    , m_stream(std::move(socket), ctx)
{
}

COcvApplicationServerHttpSessionSSL::~COcvApplicationServerHttpSessionSSL()
{
}

beast::ssl_stream<tcp::socket> & COcvApplicationServerHttpSessionSSL::Stream()
{
    return m_stream;
}

beast::ssl_stream<tcp::socket> COcvApplicationServerHttpSessionSSL::ReleaseStream()
{
    return std::move(m_stream);
}

void COcvApplicationServerHttpSessionSSL::DoTimeout()
{
    // If this is true it means we timed out performing the shutdown
    if(m_eof)
        return;

    // Start the timer again
    m_timer.expires_at(
        (std::chrono::steady_clock::time_point::max)());
    OnTimer({});
    DoEof();
}

std::string COcvApplicationServerHttpSessionSSL::GetRemoteAddress()
{
    return Stream().next_layer().remote_endpoint().address().to_string();
}

void COcvApplicationServerHttpSessionSSL::MakeWebSocketSession(boost::beast::http::request<boost::beast::http::string_body> req)
{
    std::make_shared<CApplicationServerWebSocketSessionSSL>(
        std::move(m_stream), m_state)->Run(std::move(req));
}

void COcvApplicationServerHttpSessionSSL::Run()
{
    // Make sure we run on the strand
    if(!m_strand.running_in_this_thread())
        return boost::asio::post(
            boost::asio::bind_executor(
                m_strand,
                std::bind(
                    &COcvApplicationServerHttpSessionSSL::Run,
                    shared_from_this())));

    // Run the timer. The timer is operated
    // continuously, this simplifies the code.
    OnTimer({});

    // Set the timer
    m_timer.expires_after(std::chrono::seconds(OCV_HTTP_SESSION_TIMER_EXPIRE_AFTER));

    // Perform the SSL handshake
    // Note, this is the buffered version of the handshake.
    m_stream.async_handshake(
        ssl::stream_base::server,
        buffer_.data(),
        boost::asio::bind_executor(
            m_strand,
            std::bind(
                &COcvApplicationServerHttpSessionSSL::OnHandshake,
                shared_from_this(),
                std::placeholders::_1,
                std::placeholders::_2)));
}

void COcvApplicationServerHttpSessionSSL::OnHandshake(boost::system::error_code ec, std::size_t bytes_used)
{
    // Happens when the handshake times out
    if(ec == boost::asio::error::operation_aborted)
        return;

    if(ec)
        return fail(ec, "handshake", m_Logger);

    // Consume the portion of the buffer used by the handshake
    buffer_.consume(bytes_used);

    DoRead();
}

void COcvApplicationServerHttpSessionSSL::OnShutdown(boost::system::error_code ec)
{
    // Happens when the shutdown times out
    if(ec == boost::asio::error::operation_aborted || ec == boost::asio::ssl::error::stream_truncated)
        return;

    if(ec)
        return fail(ec, "shutdown HTTPS", m_Logger);

    // At this point the connection is closed gracefully
}


void COcvApplicationServerHttpSessionSSL::OnTimer(boost::system::error_code ec)
{
    if(ec && ec != boost::asio::error::operation_aborted)
            return fail(ec, "timer", m_Logger);

        // Check if this has been upgraded to Websocket
        if(m_timer.expires_at() == (std::chrono::steady_clock::time_point::min)())
            return;

        // Verify that the timer really expired since the deadline may have moved.
        if(m_timer.expiry() <= std::chrono::steady_clock::now())
            return DoTimeout();

        // Wait on the timer
        m_timer.async_wait(
            boost::asio::bind_executor(
                m_strand,
                std::bind(
                    &COcvApplicationServerHttpSessionSSL::OnTimer,
                    shared_from_this(),
                    std::placeholders::_1)));
}

void COcvApplicationServerHttpSessionSSL::ReadRequest()
{
    // Read a request
    http::async_read(
        Stream(),
        buffer_,
        *parser_,
        boost::asio::bind_executor(
            m_strand,
            std::bind(
                &CApplicationServerBaseHttpSession::OnRead,
                shared_from_this(),
                std::placeholders::_1)));
}

void COcvApplicationServerHttpSessionSSL::WriteRequestStringBody(boost::beast::http::response<boost::beast::http::string_body> & msg)
{
    boost::beast::http::async_write(
        Stream(),
        msg,
        boost::asio::bind_executor(
            m_strand,
            std::bind(
                &CApplicationServerBaseHttpSession::OnWrite,
                shared_from_this(),
                std::placeholders::_1,
                msg.need_eof()
            )
        )
    );
}

void COcvApplicationServerHttpSessionSSL::WriteRequestFileBody(boost::beast::http::response<boost::beast::http::file_body> & msg)
{
    boost::beast::http::async_write(
        Stream(),
        msg,
        boost::asio::bind_executor(
            m_strand,
            std::bind(
                &CApplicationServerBaseHttpSession::OnWrite,
                shared_from_this(),
                std::placeholders::_1,
                msg.need_eof()
            )
        )
    );
}

void COcvApplicationServerHttpSessionSSL::DoEof()
{
    m_eof = true;

    // Set the timer
    m_timer.expires_after(std::chrono::seconds(OCV_HTTP_SESSION_TIMER_EXPIRE_DO_EOF));

    // Perform the SSL shutdown
    m_stream.async_shutdown(
        boost::asio::bind_executor(
            m_strand,
            std::bind(
                &COcvApplicationServerHttpSessionSSL::OnShutdown,
                shared_from_this(),
                std::placeholders::_1)));
}

The Visual Leak Detector gives me the following:

c:\openssl-1.1.1b\crypto\mem.c (233): abc.exe!CRYPTO_zalloc
c:\openssl-1.1.1b\crypto\err\err.c (716): abc.exe!ERR_get_state + 0x17 bytes
c:\openssl-1.1.1b\crypto\err\err.c (443): abc.exe!ERR_clear_error + 0x5 bytes
c:\usr\work\abc_repo\ext\boost_1_68_0\boost\asio\ssl\detail\impl\engine.ipp (235): abc.exe!boost::asio::ssl::detail::engine::perform
c:\usr\work\abc_repo\ext\boost_1_68_0\boost\asio\ssl\detail\impl\engine.ipp (137): abc.exe!boost::asio::ssl::detail::engine::handshake
c:\usr\work\abc_repo\ext\boost_1_68_0\boost\asio\ssl\detail\buffered_handshake_op.hpp (70): abc.exe!boost::asio::ssl::detail::buffered_handshake_op<boost::asio::const_buffer>::process<boost::asio::const_buffer const * __ptr64> + 0x1F bytes
c:\usr\work\abc_repo\ext\boost_1_68_0\boost\asio\ssl\detail\buffered_handshake_op.hpp (48): abc.exe!boost::asio::ssl::detail::buffered_handshake_op<boost::asio::const_buffer>::operator()
c:\usr\work\abc_repo\ext\boost_1_68_0\boost\asio\ssl\detail\io.hpp (136): abc.exe!boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>,boost::asio::ssl::detail::buffered_handshake_op<boost::asio::const_buffer>,boost::asio::executor_binder<std::_Binder<std::_Unforced,void (__cdecl CabcApplicationServerH + 0x50 bytes
c:\usr\work\abc_repo\ext\boost_1_68_0\boost\asio\ssl\detail\io.hpp (333): abc.exe!boost::asio::ssl::detail::async_io<boost::asio::basic_stream_socket<boost::asio::ip::tcp>,boost::asio::ssl::detail::buffered_handshake_op<boost::asio::const_buffer>,boost::asio::executor_binder<std::_Binder<std::_Unforced,void (__cdecl CabcApplicationServ + 0x87 bytes
c:\usr\work\abc_repo\ext\boost_1_68_0\boost\asio\ssl\stream.hpp (505): abc.exe!boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp> >::async_handshake<boost::asio::const_buffer,boost::asio::executor_binder<std::_Binder<std::_Unforced,void (__cdecl CabcApplicationServerHttpSessionSSL::*)(boost::system::erro + 0x5E bytes
c:\usr\work\abc_repo\ext\boost_1_68_0\boost\beast\experimental\core\ssl_stream.hpp (485): abc.exe!boost::beast::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp> >::async_handshake<boost::asio::const_buffer,boost::asio::executor_binder<std::_Binder<std::_Unforced,void (__cdecl CabcApplicationServerHttpSessionSSL::*)(boost::system::erro
c:\usr\work\abc_repo\util\capplicationserverhttpsession.cpp (343): abc.exe!CabcApplicationServerHttpSessionSSL::Run + 0x154 bytes

In some of the Leaks I also have:

c:\usr\work\abc_repo\ext\boost_1_68_0\boost\asio\ssl\detail\impl\engine.ipp (290): abc.exe!boost::asio::ssl::detail::engine::do_accept

Of course seems related to the ssl handshake but I check the session shutdown and seems ok.

Thank you in advance.

Answer 1

Every thread that uses async_handshake() leaks memory. I added OPENSSL_thread_stop() at the end of my thread procedure and it solved the issue.

Took it from here: https://github.com/openssl/openssl/issues/3033#issuecomment-289838302