How to use boost::beast, download a file no blocking and with responses

Question

I have started with this example so won't post all the code. My objective is to download a large file without blocking my main thread. The second objective is to get notifications so I can update a progress bar. I do have the code working a couple of ways. First is to just ioc.run(); and let it go to work, I get the file downloaded. But I can not find anyway to start the session without blocking.

The second way I can make the calls down to http::async_read_some and the call works but I can not get a response that I can use. I don't know if there is a way to pass a lambda that captures.

The #if 0..#else..#endif switches the methods. I'm sure there is a simple way but I just can not see it. I'll clean up the code when I get it working, like setting the local file name. Thanks.

    std::size_t on_read_some(boost::system::error_code ec, std::size_t bytes_transferred)
    {
        if (ec);//deal with it... 
        if (!bValidConnection) {
            std::string_view view((const char*)buffer_.data().data(), bytes_transferred);
            auto pos = view.find("Content-Length:");
            if (pos == std::string_view::npos)
                ;//error
            file_size = std::stoi(view.substr(pos+sizeof("Content-Length:")).data());
            if (!file_size)
                ;//error
            bValidConnection = true;
        }
        else {
            file_pos += bytes_transferred;
            response_call(ec, file_pos);
        }
#if 0
        std::cout << "in on_read_some caller\n";
        http::async_read_some(stream_, buffer_, file_parser_, std::bind(
            response_call,
            std::placeholders::_1,
            std::placeholders::_2));
#else
        std::cout << "in on_read_some inner\n";
        http::async_read_some(stream_, buffer_, file_parser_, std::bind(
            &session::on_read_some,
            shared_from_this(),
            std::placeholders::_1,
            std::placeholders::_2));
#endif
        return buffer_.size();
    }

The main, messy but.....

struct lambda_type {
    bool bDone = false;
    void operator ()(const boost::system::error_code ec, std::size_t bytes_transferred) {
        ;
    }
};
int main(int argc, char** argv)
{
    auto const host = "reserveanalyst.com";
    auto const port = "443";
    auto const target = "/downloads/demo.msi";
    int version = argc == 5 && !std::strcmp("1.0", argv[4]) ? 10 : 11;

    boost::asio::io_context ioc;
    ssl::context ctx{ ssl::context::sslv23_client };

    load_root_certificates(ctx);
    //ctx.load_verify_file("ca.pem");

    auto so = std::make_shared<session>(ioc, ctx);
    so->run(host, port, target, version);

    bool bDone = false;
    auto const lambda = [](const boost::system::error_code ec, std::size_t bytes_transferred) {
        std::cout << "data lambda bytes: " << bytes_transferred << " er: " << ec.message() << std::endl;
    };

    lambda_type lambda2;
    so->set_response_call(lambda);
    ioc.run();

    std::cout << "not in ioc.run()!!!!!!!!" << std::endl;

    so->async_read_some(lambda);

    //pseudo message pump when working.........
    for (;;) {
        std::this_thread::sleep_for(250ms);
        std::cout << "time" << std::endl;
    }
    return EXIT_SUCCESS;
}

And stuff I've added to the class session

class session : public std::enable_shared_from_this<session>
{
        using response_call_type = void(*)(boost::system::error_code ec, std::size_t bytes_transferred);
        http::response_parser<http::file_body> file_parser_;
        response_call_type response_call;
        //
        bool bValidConnection = false;
        std::size_t file_pos = 0;
        std::size_t file_size = 0;
    
    public:
        auto& get_result() { return res_; }
        auto& get_buffer() { return buffer_; }
        void set_response_call(response_call_type the_call) { response_call = the_call; }

Answer 1

I've updated this as I finally put it to use and I wanted the old method where I could download to a file or a string. Link to how asio works, great talk.

CppCon 2016 Michael Caisse Asynchronous IO with BoostAsio

As for my misunderstanding of how to pass a lambda, here is Adam Nevraumont's answer

There are two ways to compile this using a type to select the method. Both are shown at the beginning of main. You can construct either a file downloader or string downloader by selecting the type of beast parser. The parsers don't have the same constructs so an if constexpr compile time conditions are used. And I checked, a release build of the downloader is about 1K so pretty light weight for what it does. In the case of a small string you don't have to handle the call backs. either pass an empty lambda or add the likes of:

if(response_call)
    response_call(resp_ok, test);

This looks to be a pretty clean way to get the job done so I've updated this post as of 11/27/2202.

The code:

//
// Copyright (c) 2016-2019 Vinnie Falco (vinnie dot falco at gmail dot com)
//
// Distributed under the Boost Software License, Version 1.0. (See accompanying
// file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
//
// Official repository: https://github.com/boostorg/beast

//------------------------------------------------------------------------------
//
// Example: HTTP SSL client, synchronous, usable in a thread with a message pump
// Added code to use from a message pump
// Also useable as body to a file download, or body to string
//
//------------------------------------------------------------------------------

#include <boost/beast/core.hpp>
#include <boost/beast/http.hpp>
#include <boost/beast/ssl.hpp>
#include <boost/beast/version.hpp>
#include <boost/asio/connect.hpp>
#include <boost/asio/ip/tcp.hpp>
#include <boost/asio/ssl/error.hpp>
#include <boost/asio/ssl/stream.hpp>
#include <cstdlib>
#include <iostream>
#include <string>
#include <fstream>

//the boost shipped certificates
#include <boost/../libs/beast/example/common/root_certificates.hpp>

//TODO add your ssl libs as you would like
#ifdef _M_IX86
#pragma comment(lib, "libcrypto.lib")
#pragma comment(lib, "libssl.lib")
#elif _M_X64
#pragma comment(lib, "libcrypto-3-x64.lib")
#pragma comment(lib, "libssl-3-x64.lib")
#endif

namespace downloader {
    namespace beast = boost::beast; // from <boost/beast.hpp>
    namespace http = beast::http;   // from <boost/beast/http.hpp>
    namespace net = boost::asio;    // from <boost/asio.hpp>
    namespace ssl = net::ssl;       // from <boost/asio/ssl.hpp>
    using tcp = net::ip::tcp;       // from <boost/asio/ip/tcp.hpp>

    //specialization if using < c++17; see both 'if constexpr' below.
    //this is not needed otherwise
    //namespace detail {
    //    template<typename Type>
    //    void open_file(http::parser < false, Type>& p, const char* name, boost::system::error_code& file_open_ec) { }
    //    template<>
    //    void open_file(http::parser<false, http::file_body>& p, const char* name, boost::system::error_code& file_open_ec) {
    //        p.get().body().open(name, boost::beast::file_mode::write, file_open_ec);
    //    }
    //    template<typename Type>
    //    std::string get_string(http::parser < false, Type>& p) { return std::string{}; }
    //    template<>
    //    std::string get_string(http::parser<false, http::string_body>& p) {
    //        return p.get().body();
    //    }
    //} //namespace detail

    enum responses {
        resp_null,
        resp_ok,
        resp_done,
        resp_error,
    };
    using response_call_type = std::function< void(responses, std::size_t)>;
 
    template<typename ParserType>
    struct download {
        //as these can be set with array initialization
        const char* target_ = "/";
        const char* filename_ = "test.txt";
        const char* host_ = "lakeweb.net";

        std::string body_;
        using response_call_type = std::function< void(responses, std::size_t)>;
        response_call_type response_call;

        boost::asio::io_context     ioc_;
        ssl::context                ctx_{ ssl::context::sslv23_client };
        ssl::stream<tcp::socket>    stream_{ ioc_, ctx_ };
        tcp::resolver               resolver_{ ioc_ };
        boost::beast::flat_buffer   buffer_;
        uint64_t                     file_size_{};
        int                         version{ 11 };

        void set_response_call(response_call_type the_call) { response_call = the_call; }
        uint64_t get_file_size() { return file_size_; }
        void stop() { ioc_.stop(); }
        bool stopped() { return ioc_.stopped(); }
        std::string get_body() { return std::move(body_); }
        void run() {
            try {
                // TODO should have a timer in case of a hang
                load_root_certificates(ctx_);

                // Set SNI Hostname (many hosts need this to handshake successfully)
                if (!SSL_set_tlsext_host_name(stream_.native_handle(), host_)) {
                    boost::system::error_code ec{ static_cast<int>(::ERR_get_error()), boost::asio::error::get_ssl_category() };
                    throw boost::system::system_error{ ec };
                }
                //TODO resolve is depreciated, use endpoint
                auto const results = resolver_.resolve(host_, "443");

                boost::asio::connect(stream_.next_layer(), results.begin(), results.end());
                stream_.handshake(ssl::stream_base::client);

                // Set up an HTTP GET request message
                http::request<http::string_body> req{ http::verb::get, target_, version };
                req.set(http::field::host, host_);
                req.set(http::field::user_agent, "mY aGENT");

                // Send the HTTP request to the remote host
                http::write(stream_, req);

                // Read the header
                boost::system::error_code file_open_ec;
                http::parser<false, ParserType> p;
                p.body_limit((std::numeric_limits<std::uint32_t>::max)());

                //detail::open_file(p, filename_, file_open_ec);
                //or => c++17
                if constexpr (std::is_same_v<ParserType, http::file_body>)
                    p.get().body().open(filename_, boost::beast::file_mode::write, file_open_ec);

                http::read_header(stream_, buffer_, p);
                file_size_ = p.content_length().has_value() ? p.content_length().value() : 0;

                //Read the body
                uint64_t test{};
                boost::system::error_code rec;
                for (;;) {
                    test += http::read_some(stream_, buffer_, p, rec);
                    if (test >= file_size_) {
                        response_call(resp_done, 0);
                        break;
                    }
                    response_call(resp_ok, test);
                }

                // Gracefully close the stream
                boost::system::error_code ec;
                stream_.shutdown(ec);
                if (ec == boost::asio::error::eof)
                {
                    // Rationale:
                    // http://stackoverflow.com/questions/25587403/boost-asio-ssl-async-shutdown-always-finishes-with-an-error
                    ec.assign(0, ec.category());
                }
                if (ec)
                    throw boost::system::system_error{ ec };

                //value = detail::get_string(p);
                //or => c++17
                if constexpr (std::is_same_v<ParserType, http::string_body>)
                    body_ = p.get().body();
            }
            catch (std::exception const& e)
            {
                std::cerr << "Error: " << e.what() << std::endl;
                response_call(resp_error, -1);
            }
            ioc_.stop();
        }
    };
}//namespace downloadns

//comment to test with string body
#define THE_FILE_BODY_TEST
int main(int argc, char** argv)
{
    using namespace downloader;

#ifdef THE_FILE_BODY_TEST
    download<http::file_body> dl{"/Nasiri%20Abarbekouh_Mahdi.pdf", "test.pdf"};
#else //string body test
    download<http::string_body> dl{ "/robots.txt" };
#endif

    responses dl_response{ resp_null };
    size_t cur_size{};

    auto static const lambda = [&dl_response, &dl, &cur_size](responses response, std::size_t bytes_transferred) {
        if ((dl_response = response) == resp_ok) {
            cur_size += bytes_transferred;
            size_t sizes = dl.get_file_size() - cur_size;//because size is what is left
            //drive your progress bar from here in a GUI app
        }
    };
    dl.set_response_call(lambda);
    std::thread thread{ [&dl]() { dl.run(); } };

    //thread has started, now the pseudo message pump
    bool quit = false; //true: as if a cancel button was pushed; won't finish download
    for (int i = 0; ; ++i) {

        switch (dl_response) { //ad hoc as if messaged
        case resp_ok:
            std::cout << "from sendmessage: " << cur_size << std::endl;
            dl_response = resp_null;
            break;
        case resp_done:
            std::cout << "from sendmessage: done" << std::endl;
            dl_response = resp_null;
            break;
        case resp_error:
            std::cout << "from sendmessage: error" << std::endl;
            dl_response = resp_null;
        }//switch

        if (!(i % 5))
            std::cout << "in message pump, stopped: " << std::boolalpha << dl.stopped() << std::endl;

        std::this_thread::sleep_for(std::chrono::milliseconds(100));
        if (quit && i == 10) //the cancel message
            dl.stop();
        if (!(i % 20) && dl.stopped()) {//dl job was quit or error or finished
            std::cout << "dl is stopped" << std::endl;
            break;
        }
    }
#ifdef THE_FILE_BODY_TEST
    std::cout << "file written named: 'test.txt'" << std::endl;
#else
    std::string res = dl.get_body();
    std::cout << "body retrieved:\n" << res << std::endl;
#endif
    if (thread.joinable())//in the case a thread was never started
        thread.join();
    std::cout << "exiting, program all done" << std::endl;

    return EXIT_SUCCESS;
}

Answer 2

I strongly recommend against using the low-level [async_]read_some function instead of using http::[async_]read as intended with http::response_parser<http::buffer_body>

I do have an example of that - which is a little bit complicated by the fact that it also uses Boost Process to concurrently decompress the body data, but regardless it should show you how to use it:

How to read data from Internet using muli-threading with connecting only once?

I guess I could tailor it to your specific example given more complete code, but perhaps the above is good enough? Also see "Relay an HTTP message" in libs/beast/example/doc/http_examples.hpp which I used as "inspiration".

Caution: the buffer arithmetic is not intuitive. I think this is unfortunate and should not have been necessary, so pay (very) close attention to these samples for exactly how that's done.