LMDB: How to get back to a valid state after an MDB_MAP_FULL error

Question

I'm using LMDB to log time series data in my application. Entries are added to the database at regular intervals or at specific events. I'm adding mechanisms to clear old entries to avoid that the database grows out of proportion. However, I want to be able to handle the case where the map size limit is reached before any cleanup was performed (MDB_MAP_FULL).

My problem is that, once I reach MDB_MAP_FULL, I get MDB_MAP_FULL errors also when deleting entries. That is, I cannot get back to a valid state by removing entries from the database.

Increasing the environment mapsize after getting MDB_MAP_FULL seems to fix the problem, but I don't want to have to increase the mapsize every time i reach the size limit. Also, this would cause thread safety issues since mdb_env_set_mapsize cannot be called if there are active transactions in the same process.

Is there a way to get back to a valid state without increasing the mapsize? Am I doing something wrong in my code to leave the database in an invalid state?

The following code snippet reproduces my problem. I add entries to the database until i get an MDB_MAP_FULL error. Then, I try to remove all the entries from the database. After getting MDB_MAP_FULL I can delete some entries but eventually I get MDB_MAP_FULL again.

    // Create environment
    MDB_env* env;
    int ec = mdb_env_create(&env);
    if (ec != MDB_SUCCESS) {
        throw Lmdb::LmdbError("Unable to create database environment", ec);
    }

    // Set mapsize
    size_t maxSizeInBytes = 20*boost::interprocess::mapped_region::get_page_size();
    ec = mdb_env_set_mapsize(env, maxSizeInBytes);
    if (ec != MDB_SUCCESS) {
        throw Lmdb::LmdbError("Unable to configure database environment memory map size", ec);
    }

    // Open environment
    ec = mdb_env_open(env, dbPath.c_str(), 0, 0644);
    if (ec != MDB_SUCCESS) {
        throw Lmdb::LmdbError("Unable to open database environment", ec);
    }

    // Open database
    MDB_dbi dbi;
    {
        MDB_txn* txn;
        ec = mdb_txn_begin(env, nullptr, 0, &txn);
        if (ec != MDB_SUCCESS) {
            throw Lmdb::LmdbError("Unable to start database transaction", ec);
        }
        ec = mdb_dbi_open(txn, nullptr, MDB_INTEGERKEY, &dbi);
        if (ec != MDB_SUCCESS) {
            throw Lmdb::LmdbError("Unable to open database", ec);
        }
        ec = mdb_txn_commit(txn);
        if (ec != MDB_SUCCESS) {
            throw Lmdb::LmdbError("Unable to commit database transaction", ec);
        }
    }

    // Fill DB
    size_t elementsAdded = 0;
    size_t maxCount = 1000000; // Set limit to ensure test does not hang in case of some error
    for (size_t i = 0; i < maxCount; i++) {
        MDB_txn* txn;
        ec = mdb_txn_begin(env, nullptr, 0, &txn);
        if (ec != MDB_SUCCESS) {
            throw Lmdb::LmdbError("Unable to start database transaction", ec);
        }

        int value = std::rand();
        MDB_val db_key{sizeof(size_t), (void*)(&i)};
        MDB_val db_data{sizeof(int), (void*)(&value)};
        ec = mdb_put(txn, dbi, &db_key, &db_data, 0);
        if (ec != MDB_SUCCESS) {
            mdb_txn_abort(txn);
            throw Lmdb::LmdbError("Unable to add database entry", ec);
        }

        ec = mdb_txn_commit(txn);
        if (ec == MDB_MAP_FULL) {
            elementsAdded = i;
            SqPrintMessage("Reached max size on put commit at index: %d", i);
            break;
        } else if (ec != MDB_SUCCESS) {
            throw Lmdb::LmdbError("Unable to commit database transaction", ec);
        }
    }

    // Attempt deleting entries
    for (size_t i = 0; i < elementsAdded; i++) {
        MDB_txn* txn;
        ec = mdb_txn_begin(env, nullptr, 0, &txn);
        if (ec != MDB_SUCCESS) {
            throw Lmdb::LmdbError("Unable to start database transaction", ec);
        }

        MDB_val db_key{sizeof(size_t), (void*)(&i)};
        ec = mdb_del(txn, dbi, &db_key, nullptr);
        if (ec != MDB_SUCCESS) {
            mdb_txn_abort(txn);
            throw Lmdb::LmdbError("Unable to delete database entry", ec);
        }

        ec = mdb_txn_commit(txn);
        if (ec == MDB_MAP_FULL) {
            SqPrintMessage("Reached max size on delete commit at index: %d", i);
            break;
        } else if (ec != MDB_SUCCESS) {
            throw Lmdb::LmdbError("Unable to commit database transaction", ec);
        }
    }

    // Close environment
    mdb_env_close(env);

Output:

Reached max size on put commit at index: 1657
Reached max size on delete commit at index: 137

Adding

    ec = mdb_env_set_mapsize(env, maxSizeInBytes + 2*boost::interprocess::mapped_region::get_page_size());
    if (ec != MDB_SUCCESS) {
        throw Lmdb::LmdbError("Unable to configure database environment memory map size", ec);
    }

after the first loop I do not get MDB_MAP_FULL when deleting the elements.

Answer 1

I'm not sure exactly why you're getting MDB_MAP_FULL after deleting records, but my guess is that LMDB is trying to allocate pages to maintain its "free-list" data-structure (this is a list of unused pages which is itself stored in an LMDB DB). If it can't get contiguous pages needed to store the free list entries, it will try to get them by expanding the map.

Since, as you mention, it can be difficult to increase the map size of a DB without halting a running system, it is generally better to start with a very large map size. Fortunately, on most systems (except possibly windows?) this does not actually create a file of this size so usually applications set the map size to a very large value (100GB, or more).