mremap will not expand past size of one page

Question

I'm creating a template class with a dynamic array that behaves like std::vector but the underlying array is stored in shared memory so that it can be shared between processes. At this time, there is no synchronization built in, just working on getting the memory mapping resizing to work as intended and the code is a minimal example to show where the error is coming from

The issue I'm having is that once I resize the memory passed one page size, if I try to access the second page I get a SIGBUS error. If I allocate a larger page size with mmap, for example I've tried up to 1 MB, it will allocate that amount fine but if I try to resize that mapping larger I will again get a SIGBUS error if I try to access past the original bounds of the mapping

template<typename T>
class DistributedVector {
private:
    T* create_shared_memory(size_t size) {
        int protection = PROT_READ | PROT_WRITE;
        int visibility = MAP_SHARED;

        m_shm_fd = memfd_create("shm", 0);
        ftruncate(m_shm_fd, size);
        void* result = mmap(nullptr, size * sizeof(T), protection, visibility, m_shm_fd, 0);
        if (result == MAP_FAILED){
            std::cerr << "Mapping failed!\n";
        }
        file_size = lseek(m_shm_fd, 0, SEEK_END);
        return (T*)result;
    }

    void resize_shared_memory(T* addr, size_t size, size_t new_size){
        auto temp = create_shared_memory(new_size);
        if (temp == MAP_FAILED) {
            std::cerr << "Mapping failed!\n";
        }
        memcpy(temp, m_begin, size*sizeof(T));
        munmap(m_begin, size);
        m_begin = temp;
        m_end = m_begin + size;
        m_end_cap = m_begin + new_size;
    }

public:
    DistributedVector() {
        m_begin = create_shared_memory(INITIAL_VEC_CAPACITY);
        m_end_cap = m_begin + INITIAL_VEC_CAPACITY;
        m_end = m_begin;
    }

    size_t size() {
        return m_end - m_begin;
    }

    size_t capacity() {
        return m_end_cap - m_begin;
    }

    T* data(){
        return m_begin;
    }

    T at(size_t index){
        if (index < (m_end - m_begin)) {
            return *(m_begin+index);
        } else {
            return -1;
        }
    }

    T push_back(T ele){
        if (m_end == m_end_cap){
            resize_shared_memory(m_begin,capacity(), 2*capacity());
        }
        *m_end = ele;
        m_end++;
        return *(m_end-1);
    }

private:
    T* m_begin;
    T* m_end;
    T* m_end_cap;
    int m_shm_fd;
    size_t file_size;

};

Running this test:

std::vector<int> v;
dtl::DistributedVector<int> d;

assert(v.size() == d.size());

for(int i = 0; i < 20000000; ++i){
    v.push_back(i); 
    d.push_back(i);
    auto v_i = v.at(i);
    auto d_i = d.at(i);
    assert(v_i == d_i);
    if (i % 100 == 0)
        std::cout << v_i << ":" << d_i << '\n';
}
std::cout << '\n';

assert(v.size() == d.size());

Not seeing anything in the documentation about a limit to mremap page size, and if I manually allocate a new memory section with mmap, copy the data over, and munmap the old page, it works fine. Is there something in mremap that I am overlooking?

edit: updated code to work with memfd_create, ftruncate, and resize by mmap a new memory area and munmap the old area, and now the code works as intended

Answer 1

mremap cannot resize mapping that is both shared and anonymous.

Every virtual address region has an optional file object associated with it that is used to fetch new pages and writeback old modified ones. Since anonymous mappings have no such file, there is nowhere to store pages when you shrink it and nowhere to get more pages when you expand it. Kernel generates SIGBUS when thread is accessing file out of range via mapping - happens when file was shrunk after it was mapped.

In theory kernel could do something to make it work, but so far it doesn't and it would still cause problems with how such mapping should behave as well as complexities with tracking used portions of that mapping.

What you need is to provide a file to back your mapping. To resize it you will have to first resize the file, then remap the mapping. If you do not want to reserve a file on disk, use memfd_create to create anonymous file in memory - this way you also get an fd which can be send to another process through a Unix socket.

Answer 2

I don't think it is possible to make this work as you expect. Imagine you have two processes sharing the initial memory mapping. When one decides to remap in order to double the capacity, and if mremap() has to move (MREMAP_MAYMOVE) the address because not enough contiguous virtual address space is available right after, then the process which decided this resize operation obtains the new virtual address for this bigger mapping. But the same problem of moving the bigger mapping to a new address could arise for the other process too; nothing guaranties there is enough contiguous virtual address space on its side either. How could the other process be aware of the fact that its old mapping becomes invalid and the new one should be used instead?

I don't know why this does not fail immediately (with MAP_FAILED) but if you want your shared container to grow in many processes you certainly need an applicative protocol (with appropriate synchronisation) to inform each of these processes that some new mappings have to be considered. Note that the problem would have been the same with mmap()/copy/munmap().

By the way, return *m_end; at the end of push_back() is erroneous, since it's past-the-end (and even past the end of the memory region when we are just before the resize).

Answer 3

While I do not have a direct answer to why your code fails to remap memory (could be that MAP_ANONYMOUS + MAP_SHARED + remap do not quite work), you have a bigger issue in your code, and when you fix this issue, you will fix your mremap as well.

The biggest problem is that mremap is going to perform bit-wise copy of memory region, leaving old data behind. For templated vectors, this is unacceptable. Unless your T is both trivially copyable and trivially destructible, you have to do what normal vector does (which never uses realloc).

You need to allocate a completely new shared memory segment of appropriate size, copy (or move if no-throw movable) elements from previous segment to another and than destruct elements in the old segment.

Having said all that, it is not immediately obvious why you even do your own vector class. Why don't you use standard vector with shared memory allocator?