How to put file in boost::interprocess::managed_shared_memory?

Question

How to put a file with an arbitrary name and arbitrary size into a boost::interprocess::managed_shared_memory?

Note, I donot mean boost::interprocess::managed_mapped_file or boost::interprocess::file_mapping.

I chose managed_shared_memory because other options require a fixed file name to be specified but I need to transfer files with different names.

I need to use boost, not Win32 API.

I rummaged through a huge amount of information on the Internet, but did not find anything suitable.

Therefore, I am asking you for help. I would be very grateful to you.

Answer 1

UPDATE

Added bonus versions at the end. Now this answer presents three complete versions of the code:

1. Using managed_shared_memory as requested
2. Using message_queue as a more natural appraoch for upload/transfter
3. Using TCP sockets (Asio) as to demonstrate the flexibilities of that

All of these are using Boost only

Shared memory managed segments contain arbitrary objects. So you define an object like

 struct MyFile {
     std::string _filename;
     std::vector<char> _contents;
 };

And store it there. But, wait, not so quick, because these can only be stored safely with interprocess allocators, so adding some magic sauce (a.k.a lots of interesting typedefs to get the allocators declared, and some constructors):

namespace Shared {
    using Mem = bip::managed_shared_memory;
    using Mgr = Mem::segment_manager;

    template <typename T>
    using Alloc = bc::scoped_allocator_adaptor<bip::allocator<T, Mgr>>;

    template <typename T> using Vector = bc::vector<T, Alloc<T>>;
    using String =
        bc::basic_string<char, std::char_traits<char>, Alloc<char>>;

    struct MyFile {
        using allocator_type = Alloc<char>;

        template <typename It>
        explicit MyFile(std::string_view name, It b, It e, allocator_type alloc)

        String _filename;
        Vector<char> _contents;
    };
}

Now you can store your files like:

Shared::Mem shm(bip::open_or_create, "shared_mem", 10ull << 30);

std::ifstream ifs("file_name.txt", std::ios::binary);
std::istreambuf_iterator<char> data_begin{ifs}, data_end{};

auto loaded = shm.find_or_construct<Shared::MyFile>("file1")(
        file.native(), data_begin, data_end,
         shm.get_segment_manager());

Note that the shared memory won't actually take 30GiB right away, even though that's what 10ull << 30 specifies. On most operating systems this will be sparesely allocated and only the pages that contain data will be commited.

Improving

You might have wondered what the scoped_allocator_adaptor was for. It doesn't seem we use it?

Well, the idea was to not use find_or_construct directly per file, but to store a Vector<MyFile so you can harness the full power of BIP allocators.

The following full demo can be invoked

• with filename arguments, which will all be loaded (if they exist as regular files)
• without arguments, which will list previously loaded files

Live On Coliru

#include <boost/interprocess/managed_shared_memory.hpp>
#include <boost/interprocess/managed_mapped_file.hpp> // for COLIRU
#include <boost/interprocess/containers/vector.hpp>
#include <boost/interprocess/containers/string.hpp>
#include <boost/container/scoped_allocator.hpp>
#include <fstream>
#include <filesystem>
#include <iostream>
#include <iomanip>

namespace bip = boost::interprocess;
namespace bc = boost::container;
namespace fs = std::filesystem;

namespace Shared {
#ifdef COLIRU
    using Mem = bip::managed_mapped_file; // managed_shared_memory not allows
#else
    using Mem = bip::managed_shared_memory;
#endif
    using Mgr = Mem::segment_manager;

    template <typename T>
    using Alloc = bc::scoped_allocator_adaptor<bip::allocator<T, Mgr>>;

    template <typename T> using Vector = bc::vector<T, Alloc<T>>;
    using String = bc::basic_string<char, std::char_traits<char>, Alloc<char>>;

    struct MyFile {
        using allocator_type = Alloc<char>;

        MyFile(MyFile&&) = default;
        MyFile(MyFile const& rhs, allocator_type alloc)
            : _filename(rhs._filename.begin(), rhs._filename.end(), alloc),
              _contents(rhs._contents.begin(), rhs._contents.end(), alloc) {}

        MyFile& operator=(MyFile const& rhs) {
            _filename.assign(rhs._filename.begin(), rhs._filename.end());
            _contents.assign(rhs._contents.begin(), rhs._contents.end());
            return *this;
        }

        template <typename It>
        explicit MyFile(std::string_view name, It b, It e, allocator_type alloc)
            : _filename(name.data(), name.size(), alloc),
              _contents(b, e, alloc) {}

        String _filename;
        Vector<char> _contents;

        friend std::ostream& operator<<(std::ostream& os, MyFile const& mf) {
            return os << "Name: " << std::quoted(mf._filename.c_str())
                      << " content size: " << mf._contents.size();
        }
    };
} // namespace Shared

int main(int argc, char** argv) {
    Shared::Mem shm(bip::open_or_create, "shared_mem", 512ull << 10);

    using FileList = Shared::Vector<Shared::MyFile>;
    auto& shared_files =
        *shm.find_or_construct<FileList>("FileList")(shm.get_segment_manager());

    if (1==argc) {
        std::cout << "Displaying previously loaded files: \n";
        for (auto& entry : shared_files)
            std::cout << entry << std::endl;
    } else {
        std::cout << "Loading files: \n";
        for (auto file : std::vector<fs::path>{argv + 1, argv + argc}) {
            if (is_regular_file(file)) {
                try {
                    std::ifstream ifs(file, std::ios::binary);
                    std::istreambuf_iterator<char> data_begin{ifs}, data_end{};

                    auto& loaded = shared_files.emplace_back(
                        file.native(), data_begin, data_end);

                    std::cout << loaded << std::endl;
                } catch (std::system_error const& se) {
                    std::cerr << "Error: " << se.code().message() << std::endl;
                } catch (std::exception const& se) {
                    std::cerr << "Other: " << se.what() << std::endl;
                }
            }
        }
    }
}

When run with

g++ -std=c++17 -O2 -Wall -pedantic -pthread main.cpp -lrt -DCOLIRU
./a.out main.cpp a.out
./a.out

Prints

Loading files: 
Name: "main.cpp" content size: 3239
Name: "a.out" content size: 175176
Displaying previously loaded files: 
Name: "main.cpp" content size: 3239
Name: "a.out" content size: 175176

BONUS

In response to the comments, I think it's worth actually comparing

Message Queue version

For comparison, here's a message queue implementation

Live On Coliru

#include <boost/interprocess/ipc/message_queue.hpp>
#include <boost/endian/arithmetic.hpp>
#include <fstream>
#include <filesystem>
#include <iostream>
#include <iomanip>

namespace bip = boost::interprocess;
namespace fs = std::filesystem;
using bip::message_queue;
static constexpr auto MAX_FILENAME_LENGH = 512; // 512 bytes max filename length
static constexpr auto MAX_CONTENT_SIZE = 512ull << 10; // 512 KiB max payload size

struct Message {
    std::vector<char> _buffer;

    using Uint32 = boost::endian::big_uint32_t;
    struct header_t {
        Uint32 filename_length;
        Uint32 content_size;
    };
    static_assert(std::is_standard_layout_v<header_t> and
                  std::is_trivial_v<header_t>);

    Message() = default;

    Message(fs::path file) {
        std::string const name = file.native();
        std::ifstream ifs(file, std::ios::binary);
        std::istreambuf_iterator<char> data_begin{ifs}, data_end{};

        _buffer.resize(header_len + name.length());
        std::copy(begin(name), end(name), _buffer.data() + header_len);
        _buffer.insert(_buffer.end(), data_begin, data_end);
        header().filename_length = name.length();
        header().content_size    = size() - header_len - name.length();
    }

    Message(char const* buf, size_t size) 
        : _buffer(buf, buf+size) {}

    static constexpr auto header_len = sizeof(header_t);
    static constexpr auto max_size =
        header_len + MAX_FILENAME_LENGH + MAX_CONTENT_SIZE;

    char const* data() const { return _buffer.data(); } 
    size_t size() const      { return _buffer.size(); } 

    header_t& header() {
        assert(_buffer.size() >= header_len);
        return *reinterpret_cast<header_t*>(_buffer.data());
    }

    header_t const& header() const {
        assert(_buffer.size() >= header_len);
        return *reinterpret_cast<header_t const*>(_buffer.data());
    }

    std::string_view filename() const { 
        assert(_buffer.size() >= header_len + header().filename_length);
        return { _buffer.data() + header_len, header().filename_length };
    }

    std::string_view contents() const {
        assert(_buffer.size() >=
                header_len + header().filename_length + header().content_size);

        return {_buffer.data() + header_len + header().filename_length,
            header().content_size};
    }

    friend std::ostream& operator<<(std::ostream& os, Message const& mf) {
        return os << "Name: " << std::quoted(mf.filename())
                  << " content size: " << mf.contents().size();
    }
};

int main(int argc, char** argv) {
    message_queue mq(bip::open_or_create, "file_transport", 10, Message::max_size);

    if (1==argc) {
        std::cout << "Receiving uploaded files: \n";
        char rawbuf [Message::max_size];
        while (true) {
            size_t n;
            unsigned prio;
            mq.receive(rawbuf, sizeof(rawbuf), n, prio);

            Message decoded(rawbuf, n);
            std::cout << "Received: " << decoded << std::endl;
        }
    } else {
        std::cout << "Loading files: \n";
        for (auto file : std::vector<fs::path>{argv + 1, argv + argc}) {
            if (is_regular_file(file)) {
                try {
                    Message encoded(file);
                    std::cout << "Sending: " << encoded << std::endl;

                    mq.send(encoded.data(), encoded.size(), 0);
                } catch (std::system_error const& se) {
                    std::cerr << "Error: " << se.code().message() << std::endl;
                } catch (std::exception const& se) {
                    std::cerr << "Other: " << se.what() << std::endl;
                }
            }
        }
    }
}

A demo:

Note that there is a filesize limit in this approach because messages have a maximum length

TCP Socket Version

Here's a TCP socket implementation.

Live On Coliru

#include <boost/asio.hpp>
#include <boost/endian/arithmetic.hpp>
#include <vector>
#include <fstream>
#include <filesystem>
#include <iostream>
#include <iomanip>

namespace fs = std::filesystem;
using boost::asio::ip::tcp;
using boost::system::error_code;
static constexpr auto MAX_FILENAME_LENGH = 512; // 512 bytes max filename length
static constexpr auto MAX_CONTENT_SIZE = 512ull << 10; // 512 KiB max payload size

struct Message {
    std::vector<char> _buffer;

    using Uint32 = boost::endian::big_uint32_t;
    struct header_t {
        Uint32 filename_length;
        Uint32 content_size;
    };
    static_assert(std::is_standard_layout_v<header_t> and
                  std::is_trivial_v<header_t>);

    Message() = default;

    Message(fs::path file) {
        std::string const name = file.native();
        std::ifstream ifs(file, std::ios::binary);
        std::istreambuf_iterator<char> data_begin{ifs}, data_end{};

        _buffer.resize(header_len + name.length());
        std::copy(begin(name), end(name), _buffer.data() + header_len);
        _buffer.insert(_buffer.end(), data_begin, data_end);
        header().filename_length = name.length();
        header().content_size    = actual_size() - header_len - name.length();
    }

    Message(char const* buf, size_t size) 
        : _buffer(buf, buf+size) {}

    static constexpr auto header_len = sizeof(header_t);
    static constexpr auto max_size =
        header_len + MAX_FILENAME_LENGH + MAX_CONTENT_SIZE;

    char const* data() const { return _buffer.data(); }
    size_t actual_size() const { return _buffer.size(); }
    size_t decoded_size() const {
        return header().filename_length + header().content_size;
    }
    bool is_complete() const {
        return actual_size() >= header_len && actual_size() >= decoded_size();
    }

    header_t& header() {
        assert(actual_size() >= header_len);
        return *reinterpret_cast<header_t*>(_buffer.data());
    }

    header_t const& header() const {
        assert(actual_size() >= header_len);
        return *reinterpret_cast<header_t const*>(_buffer.data());
    }

    std::string_view filename() const { 
        assert(actual_size() >= header_len + header().filename_length);
        return std::string_view(_buffer.data() + header_len,
                                header().filename_length);
    }

    std::string_view contents() const {
        assert(actual_size() >= decoded_size());

        return std::string_view(_buffer.data() + header_len +
                                    header().filename_length,
                                header().content_size);
    }

    friend std::ostream& operator<<(std::ostream& os, Message const& mf) {
        return os << "Name: " << std::quoted(mf.filename())
                  << " content size: " << mf.contents().size();
    }
};

int main(int argc, char** argv) {
    boost::asio::io_context ctx;
    u_int16_t port = 8989;

    if (1==argc) {
        std::cout << "Receiving uploaded files: " << std::endl;
        tcp::acceptor acc(ctx, tcp::endpoint{{}, port});

        while (true) {
            auto s = acc.accept();
            std::cout << "Connection accepted from " << s.remote_endpoint() << std::endl;

            Message msg;
            auto buf = boost::asio::dynamic_buffer(msg._buffer);
            error_code ec;
            while (auto n = read(s, buf, ec)) {
                std::cout << "(read " << n << " bytes, " << ec.message() << ")" << std::endl;

                while (msg.is_complete()) {
                    std::cout << "Received: " << msg << std::endl;
                    buf.consume(msg.decoded_size() + Message::header_len);
                }
            }
            std::cout << "Connection closed" << std::endl;
        }
    } else {
        std::cout << "Loading files: " << std::endl;
        tcp::socket s(ctx);
        s.connect(tcp::endpoint{{}, port});

        for (auto file : std::vector<fs::path>{argv + 1, argv + argc}) {
            if (is_regular_file(file)) {
                try {
                    Message encoded(file);
                    std::cout << "Sending: " << encoded << std::endl;

                    write(s, boost::asio::buffer(encoded._buffer));
               } catch (std::system_error const& se) {
                    std::cerr << "Error: " << se.code().message() << std::endl;
                } catch (std::exception const& se) {
                    std::cerr << "Other: " << se.what() << std::endl;
                }
            }
        }
    }
}

Demo:

Note how this easily scales to larger files, multiple files in a single connection and even multiple connections simultaneously if you need. It also doesn't do double buffering, which improves performance. This is why this kind of approach is much more usual than any of your other approaches.