CODEWITHSUNDEEP
databaseberkeley-dblmdb
Jeremy Salwen
Jeremy Salwen

Reputation: 8418

In memory databases with LMDB

I have a project which uses BerkelyDB as a key value store for up to hundreds of millions of small records.

The way it's used is all the values are inserted into the database, and then they are iterated over using both sequential and random access, all from a single thread.

With BerkeleyDB, I can create in-memory databases that are "never intended to be preserved on disk". If the database is small enough to fit in the BerkeleyDB cache, it will never be written to disk. If it is bigger than the cache, then a temporary file will be created to hold the overflow. This option can speed things up significantly, as it prevents my application from writing gigabytes of dead data to disk when closing the database.

I have found that the BerkeleyDB write performance is too poor, even on an SSD, so I would like to switch to LMDB. However, based on the documentation, it doesn't seem like there is an option creating a non-persistent database.

What configuration/combination of options should I use to get the best performance out of LMDB if I don't care about persistence or concurrent access at all? i.e. to make it act like an "in-memory database" with temporary backing disk storage?

Upvotes: 4

Views: 3544

Answers (2)

hyc
hyc

Reputation: 1443

Just use MDB_NOSYNC and never call mdb_env_sync() yourself. You could also use MDB_WRITEMAP in addition. The OS will still eventually flush dirty pages to disk; you can play with /proc/sys/vm/dirty_ratio etc. to control that behavior.

Upvotes: 3

Kanti Varanasi
Kanti Varanasi

Reputation: 89

From this post: https://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio/

vm.dirty_ratio is the absolute maximum amount of system memory that can be filled with dirty pages before everything must get committed to disk. When the system gets to this point all new I/O blocks until dirty pages have been written to disk.

If the dirty ratio is too small, then you will see frequent synchronous disk writes.

Upvotes: 0

Related Questions