using multiple instances of shared memory at once

Question

to transport a video-stream between the recording program and the display program (which cannot be the same) I use shared memory. To synch the access I've put together a class, which wraps a shared_memory_object, a mapped_region and an interprocess_sharable_mutex (all of boost::interprocess)

I wrote 2 construtors, one for the "Host"-side, and one for the "Client"-side. When I use my class to transport one video-stream it works perfectly. But when I try to transport two video streams there are a few problems.

First off: here is the constructor code: (first one is the Host-Constructor, 2nd one the Client one)

    template<typename T>
    SWMRSharedMemArray<T>::SWMRSharedMemArray(std::string Name, size_t length):
        ShMutexSize(sizeof(interprocess_sharable_mutex)),
        isManager(true), _length(length), Name(Name)
    {
        shared_memory_object::remove(Name.c_str());

        shm = new shared_memory_object(create_only, Name.c_str(), read_write);
        shm->truncate(ShMutexSize + sizeof(T)*length);

        region = new mapped_region(*shm, read_write);

        void *addr = region->get_address();
        mtx = new(addr) interprocess_sharable_mutex;
        DataPtr = static_cast<T*>(addr) + ShMutexSize;
    }

    template<typename T>
    SWMRSharedMemArray<T>::SWMRSharedMemArray(std::string Name) :
        ShMutexSize(sizeof(interprocess_sharable_mutex)),
        isManager(false), Name(Name)
    {
        shm = new shared_memory_object(open_only, Name.c_str(), read_write);
        region = new mapped_region(*shm, read_write);

        _length = (region->get_size() - ShMutexSize) / sizeof(T);
        void *addr = region->get_address();
        mtx = static_cast<decltype(mtx)>(addr);
        DataPtr = static_cast<T*>(addr) + ShMutexSize;
    }

On the Host-Side everything still looks fine. But on the construction for the Clients there are problems: When I compare the shm and region objects of the first and seccond instance (which have different Names ofc, but same length, and template-type) I see that a lot of members that should differ do not. The adress and the member m_filename are different as expected, but the member m_handle is the same. For region the both adresses are different, but all members are identical.

I hope someone knows whats going on. Best Regards Uzaku

Answer 1

I've not completely grokked your code, but I was struck by the archaic use of manual memory management. Whenever I see "sizeof()" in C++ I get slightly worried :)

Confusion is almost inevitable due to the lack of abstraction, and the compiler is unable to help, because you're in "Leave Me Alone - I Know What I'm Doing" land.

Concretely, this looks wrong:

DataPtr = static_cast<T *>(addr) + ShMutexSize;

This might be correct when sizeof(T)==sizeof(char) (IOW, T is a byte), but otherwise you get pointer arithmetic, meaning that you add sizeof(T) ShMutexSize times. This is definitely wrong, because you only reserved room for the size of the mutex+the element data, directly adjacent.

So you get unused space and Undefined Behavior due to indexing beyond the size of the shared memory region.

---

So, let me contrast with two samples;

1. that reduces the dependence on pointer arithmetic
2. that does away with all the manual memory management by using managed shared memory segments

1. Manual

The manual approach that doesn't quite require the same amount of pointer trickery/resource management could look like this:

LiveCompiled On Coliru

#include <boost/interprocess/shared_memory_object.hpp>
#include <boost/interprocess/mapped_region.hpp>
#include <boost/interprocess/sync/interprocess_sharable_mutex.hpp>
#include <boost/thread/lock_guard.hpp>

namespace bip = boost::interprocess;

namespace SWMR {
    static struct server_mode_t {} const/*expr*/ server_mode = server_mode_t();
    static struct client_mode_t {} const/*expr*/ client_mode = client_mode_t();

    typedef bip::interprocess_sharable_mutex mutex;
    typedef boost::lock_guard<mutex> guard;

    template <typename T, size_t N> struct SharedMemArray {
        SharedMemArray(server_mode_t, std::string const& name) 
          : isManager(true), _name(name), 
            _shm(do_create(_name.c_str())),
            _region(_shm, bip::read_write)
        {
            _data = new (_region.get_address()) data_t;
        }

        SharedMemArray(client_mode_t, std::string const& name) 
          : isManager(false), _name(name),
            _shm(do_open(_name.c_str())),
            _region(_shm, bip::read_write),
            _data(static_cast<data_t*>(_region.get_address()))
        {
            assert(sizeof(data_t) == _region.get_size());
        }

    private:
        typedef bip::shared_memory_object shm_t;
        struct data_t { 
            mutable mutex mtx;
            T DataPtr[N];
        };

        bool               isManager;
        const std::string  _name;
        shm_t              _shm;
        bip::mapped_region _region;
        data_t            *_data;

        // functions to manage the shared memory
        shm_t static do_create(char const* name) {
            shm_t::remove(name);
            shm_t result(bip::create_only, name, bip::read_write);
            result.truncate(sizeof(data_t));
            return boost::move(result);
        }

        shm_t static do_open(char const* name) {
            return shm_t(bip::open_only, name, bip::read_write);
        }

      public:
        mutex& get_mutex() const { return _data->mtx; }

        typedef T       *iterator;
        typedef T const *const_iterator;

        iterator data()                { return _data->DataPtr; }
        const_iterator data() const    { return _data->DataPtr; }

        iterator begin()               { return data(); }
        const_iterator begin() const   { return data(); }

        iterator end()                 { return begin() + N; }
        const_iterator end() const     { return begin() + N; }

        const_iterator cbegin() const  { return begin(); }
        const_iterator cend() const    { return end(); }
    };
}

#include <vector>

static const std::string APP_UUID = "61ab4f43-2d68-46e1-9c8d-31d577ce3aa7";

struct UserData {
    int   i;
    float f;
};

#include <boost/range/algorithm.hpp>
#include <boost/foreach.hpp>
#include <iostream>

int main() {
    using namespace SWMR;
    SharedMemArray<int, 20>      s_ints   (server_mode, APP_UUID + "-ints");
    SharedMemArray<float, 72>    s_floats (server_mode, APP_UUID + "-floats");
    SharedMemArray<UserData, 10> s_udts   (server_mode, APP_UUID + "-udts");

    {
        guard lk(s_ints.get_mutex());
        boost::fill(s_ints, 42);
    }

    {
        guard lk(s_floats.get_mutex());
        boost::fill(s_floats, 31415);
    }

    {
        guard lk(s_udts.get_mutex());
        UserData udt = { 42, 3.14 };
        boost::fill(s_udts, udt);
    }

    SharedMemArray<int, 20>      c_ints   (client_mode, APP_UUID + "-ints");
    SharedMemArray<float, 72>    c_floats (client_mode, APP_UUID + "-floats");
    SharedMemArray<UserData, 10> c_udts   (client_mode, APP_UUID + "-udts");

    {
        guard lk(c_ints.get_mutex());
        assert(boost::equal(std::vector<int>(boost::size(c_ints), 42), c_ints));
    }

    {
        guard lk(c_floats.get_mutex());
        assert(boost::equal(std::vector<int>(boost::size(c_floats), 31415), c_floats));
    }

    {
        guard lk(c_udts.get_mutex());
        BOOST_FOREACH(UserData& udt, c_udts)
            std::cout << udt.i << "\t" << udt.f << "\n";
    }
}

Notes

• it reuses code
• it doesn't do unnecessary dynamic allocations (which makes the class much easier to "get right" for Rule Of Three)
• it uses a data_t struct to get rid of the manual offset calculations (you can just do data->mtx or data->DataPtr)

• it adds iterator and begin()/end() definitions so that you can use the SharedMemArray directly as a range, e.g. with algorithms like boost::equal and BOOST_FOREACH:

  assert(boost::equal(some_vector, c_floats));
  
  BOOST_FOREACH(UserData& udt, c_udts)
      std::cout << udt.i << "\t" << udt.f << "\n";
  

• for now, it uses a statically known number of elements (N).

If you don't want this, I'd certainly opt for the approach that uses managed segments (under 2.) because that will take care of all the (re)allocation mechanics for you.

---

2. Using a managed_shared_memory segment

What do we use in C++ when we want dynamically sized arrays? Correct: std::vector.

Now std::vector can be taught to allocate from the shared memory, but you'll need to pass it an Boost Interprocess allocator. This allocator knows how to work with a segment_manager to perform the allocations from shared memory.

Here's a relatively straight up translation to using managed_shared_memory

LiveCompiled On Coliru

#include <boost/container/scoped_allocator.hpp>

#include <boost/container/vector.hpp>
#include <boost/container/string.hpp>

#include <boost/interprocess/allocators/allocator.hpp>
#include <boost/interprocess/managed_shared_memory.hpp>
#include <boost/interprocess/offset_ptr.hpp>
#include <boost/interprocess/sync/interprocess_sharable_mutex.hpp>
#include <boost/thread/lock_guard.hpp>

namespace Shared {
    namespace bip = boost::interprocess;
    namespace bc  = boost::container;

    using shm_t = bip::managed_shared_memory;
    using mutex = bip::interprocess_sharable_mutex;
    using guard = boost::lock_guard<mutex>;

    template <typename T> using allocator    = bc::scoped_allocator_adaptor<
                                                   bip::allocator<T, shm_t::segment_manager>
                                               >;
    template <typename T> using vector       = bc::vector<T, allocator<T> >;
    template <typename T> using basic_string = bc::basic_string<T, std::char_traits<T>, allocator<T> >;

    using string  = basic_string<char>;
    using wstring = basic_string<wchar_t>;
}

namespace SWMR {
    namespace bip = boost::interprocess;

    static struct server_mode_t {} const/*expr*/ server_mode = server_mode_t();
    static struct client_mode_t {} const/*expr*/ client_mode = client_mode_t();

    template <typename T> struct SharedMemArray {

    private:
        struct data_t {
            using allocator_type = Shared::allocator<void>;

            data_t(size_t N, allocator_type alloc) : elements(alloc) { elements.resize(N); }
            data_t(allocator_type alloc)           : elements(alloc) {}

            mutable Shared::mutex mtx;
            Shared::vector<T> elements;
        };

        bool               isManager;
        const std::string  _name;
        Shared::shm_t      _shm;
        data_t            *_data;

        // functions to manage the shared memory
        Shared::shm_t static do_create(char const* name) {
            bip::shared_memory_object::remove(name);
            Shared::shm_t result(bip::create_only, name, 1ul << 20); // ~1 MiB
            return boost::move(result);
        }

        Shared::shm_t static do_open(char const* name) {
            return Shared::shm_t(bip::open_only, name);
        }

      public:
        SharedMemArray(server_mode_t, std::string const& name, size_t N = 0)
          : isManager(true), _name(name), _shm(do_create(_name.c_str()))
        {
            _data = _shm.find_or_construct<data_t>(name.c_str())(N, _shm.get_segment_manager());
        }

        SharedMemArray(client_mode_t, std::string const& name)
          : isManager(false), _name(name), _shm(do_open(_name.c_str()))
        {
            auto found = _shm.find<data_t>(name.c_str());
            assert(found.second);
            _data = found.first;
        }

        Shared::mutex&  mutex() const             { return _data->mtx; }
        Shared::vector<T>      & elements()       { return _data->elements; }
        Shared::vector<T> const& elements() const { return _data->elements; }
    };
}

#include <vector>

static const std::string APP_UUID = "93f6b721-1d34-46d9-9877-f967fea61cf2";

struct UserData {
    using allocator_type = Shared::allocator<void>;

    UserData(allocator_type alloc) : text(alloc) {}
    UserData(UserData const& other, allocator_type alloc) : i(other.i), text(other.text, alloc) {}
    UserData(int i, Shared::string t) : i(i), text(t) {}
    template <typename T> UserData(int i, T&& t, allocator_type alloc) : i(i), text(std::forward<T>(t), alloc) {}

    // data
    int   i;
    Shared::string text;
};

#include <boost/range/algorithm.hpp>
#include <boost/foreach.hpp>
#include <iostream>

int main() {
    using namespace SWMR;
    SharedMemArray<int>      s_ints(server_mode, APP_UUID + "-ints", 20);
    SharedMemArray<UserData> s_udts(server_mode, APP_UUID + "-udts");
    // server code

    {
        Shared::guard lk(s_ints.mutex());
        boost::fill(s_ints.elements(), 99);

        // or manipulate the vector. Any allocations go to the shared memory segment automatically
        s_ints.elements().push_back(42);
        s_ints.elements().assign(20, 42);
    }

    {
        Shared::guard lk(s_udts.mutex());
        s_udts.elements().emplace_back(1, "one");
    }

    // client code
    SharedMemArray<int>      c_ints(client_mode, APP_UUID + "-ints");
    SharedMemArray<UserData> c_udts(client_mode, APP_UUID + "-udts");

    {
        Shared::guard lk(c_ints.mutex());
        auto& e = c_ints.elements();
        assert(boost::equal(std::vector<int>(20, 42), e));
    }

    {
        Shared::guard lk(c_udts.mutex());
        BOOST_FOREACH(UserData& udt, c_udts.elements())
            std::cout << udt.i << "\t'" << udt.text << "'\n";
    }
}

Notes:

• Since you're now storing first class C++ objects, the sizes are not static. In fact, you can push_back and if the capacity is exceeded, the container will just reallocate from using the segment's allocator.

• I've elected to use C++11 for the convenience typedefs in namespace Shared. However, all of these can work in c++03, though with more verbosity

• I've also elected to use scoped allocators through out. This means that if T is a (user-defined) type that /also/ uses an allocator (e.g. all standard containers, std::deque, std::packaged_task, std::tuple etc. the allocator's segment reference will be implicitly passed to the elements when they're internally constructed. This is e.g. why the lines

  elements.resize(N);
  

  and

  s_udts.elements().emplace_back(1, "one");
  

  are able to compile without explicitly passing an allocator for the element's constructor.

• The sample UserData class exploits this to show how you can contain a std::string (or actually, a Shared::string) which magically allocates from the same memory segment as the container.

3. Bonus

Note also that this opens up the possibility to store all the containers inside a single shared_memory_object this may be beneficial, and therefore I present a variation that shows this approach:

LiveCompiled On Coliru

#include <boost/container/scoped_allocator.hpp>

#include <boost/container/vector.hpp>
#include <boost/container/string.hpp>

#include <boost/interprocess/allocators/allocator.hpp>
#include <boost/interprocess/managed_shared_memory.hpp>
#include <boost/interprocess/offset_ptr.hpp>
#include <boost/interprocess/sync/interprocess_sharable_mutex.hpp>
#include <boost/thread/lock_guard.hpp>

namespace Shared {
    namespace bip = boost::interprocess;
    namespace bc  = boost::container;

    using msm_t = bip::managed_shared_memory;
    using mutex = bip::interprocess_sharable_mutex;
    using guard = boost::lock_guard<mutex>;

    template <typename T> using allocator    = bc::scoped_allocator_adaptor<
                                                   bip::allocator<T, msm_t::segment_manager>
                                               >;
    template <typename T> using vector       = bc::vector<T, allocator<T> >;
    template <typename T> using basic_string = bc::basic_string<T, std::char_traits<T>, allocator<T> >;

    using string  = basic_string<char>;
    using wstring = basic_string<wchar_t>;
}

namespace SWMR {
    namespace bip = boost::interprocess;
    namespace bc  = boost::container;

    class Segment {
      public:
        // LockableObject, base template
        //
        // LockableObject contains a `Shared::mutex` and an object of type T
        template <typename T, typename Enable = void> struct LockableObject;

        // Partial specialization for the case when the wrapped object cannot
        // use the shared allocator: the constructor is just forwarded
        template <typename T>
        struct LockableObject<T, typename boost::disable_if<bc::uses_allocator<T, Shared::allocator<T> >, void>::type>
        {
            template <typename... CtorArgs>
            LockableObject(CtorArgs&&... args) : object(std::forward<CtorArgs>(args)...) {}
            LockableObject() : object() {}

            mutable Shared::mutex mutex;
            T object;

          private:
            friend class Segment;
            template <typename... CtorArgs>
            static LockableObject& locate_by_name(Shared::msm_t& msm, const char* tag, CtorArgs&&... args) {
                return *msm.find_or_construct<LockableObject<T> >(tag)(std::forward<CtorArgs>(args)...);
            }
        };

        // Partial specialization for the case where the contained object can
        // use the shared allocator;
        //
        // Construction (using locate_by_name) adds the allocator as the last
        // argument.
        template <typename T>
        struct LockableObject<T, typename boost::enable_if<bc::uses_allocator<T, Shared::allocator<T> >, void>::type>
        {
            using allocator_type = Shared::allocator<void>;

            template <typename... CtorArgs>
            LockableObject(CtorArgs&&... args) : object(std::forward<CtorArgs>(args)...) {}
            LockableObject(allocator_type alloc = {}) : object(alloc) {}

            mutable Shared::mutex mutex;
            T object;

          private:
            friend class Segment;
            template <typename... CtorArgs>
            static LockableObject& locate_by_name(Shared::msm_t& msm, const char* tag, CtorArgs&&... args) {
                return *msm.find_or_construct<LockableObject>(tag)(std::forward<CtorArgs>(args)..., Shared::allocator<T>(msm.get_segment_manager()));
            }
        };

        Segment(std::string const& name, size_t capacity = 1024*1024) // default 1 MiB
            : _msm(bip::open_or_create, name.c_str(), capacity)
        {
        }

        template <typename T, typename... CtorArgs>
        LockableObject<T>& getLockable(char const* tag, CtorArgs&&... args) {
            return LockableObject<T>::locate_by_name(_msm, tag, std::forward<CtorArgs>(args)...);
        }

    private:
        Shared::msm_t _msm;
    };
}

#include <vector>

static char const* const APP_UUID = "249f3878-3ddf-4473-84b2-755998952da1";

struct UserData {
    using allocator_type = Shared::allocator<void>;
    using String         = Shared::string;

    UserData(allocator_type alloc) : text(alloc) { }
    UserData(int i, String t) : i(i), text(t) { }
    UserData(UserData const& other, allocator_type alloc) : i(other.i), text(other.text, alloc) { }

    template <typename T>
        UserData(int i, T&& t, allocator_type alloc)
            : i(i), text(std::forward<T>(t), alloc)
        { }

    // data
    int i;
    String text;
};

#include <boost/range/algorithm.hpp>
#include <boost/foreach.hpp>
#include <iostream>

int main() {
    using IntVec = Shared::vector<int>;
    using UdtVec = Shared::vector<UserData>;

    boost::interprocess::shared_memory_object::remove(APP_UUID); // for demo

    // server code
    {
        SWMR::Segment server(APP_UUID);

        auto& s_ints = server.getLockable<IntVec>("ints", std::initializer_list<int> {1,2,3,4,5,6,7,42}); // allocator automatically added
        auto& s_udts = server.getLockable<UdtVec>("udts");

        {
            Shared::guard lk(s_ints.mutex);
            boost::fill(s_ints.object, 99);

            // or manipulate the vector. Any allocations go to the shared memory segment automatically
            s_ints.object.push_back(42);
            s_ints.object.assign(20, 42);
        }

        {
            Shared::guard lk(s_udts.mutex);
            s_udts.object.emplace_back(1, "one"); // allocates the string in shared memory, and the UserData element too
        }
    }

    // client code
    {
        SWMR::Segment client(APP_UUID);

        auto& c_ints = client.getLockable<IntVec>("ints", 20, 999); // the ctor arguments are ignored here
        auto& c_udts = client.getLockable<UdtVec>("udts");

        {
            Shared::guard lk(c_ints.mutex);
            IntVec& ivec = c_ints.object;
            assert(boost::equal(std::vector<int>(20, 42), ivec));
        }

        {
            Shared::guard lk(c_udts.mutex);
            BOOST_FOREACH(UserData& udt, c_udts.object)
                std::cout << udt.i << "\t'" << udt.text << "'\n";
        }
    }
}

Notes:

• you can now store anything, not just "dynamic arrays" (vector<T>). You could just do:

  auto& c_udts = client.getLockable<double>("a_single_double");
  

• when you store a container that is compatible with the shared allocator, LockableObject's construction method will transparently add the allocator instance as the last constructor argument for the contained T object;.

• I moved the remove() call out of the Segment class, making it unnecessary to distinguish between client/server mode. We just use open_or_create and find_or_construct.