CODEWITHSUNDEEP
node.jsrediskey-valuekey-value-storeleveldb
Ruben Verborgh
Ruben Verborgh

Reputation: 3665

Fastest, non-memory-based, multi-process key-value store for Node.js

What is the fastest non-memory key-value store for Node.js supporting multiple processes?

I need to store simple key-value string/string pairs (not documents or JSON, just strings).
Here are some examples (there would be millions of those):

I have tried:

A workaround for LevelDB is multilevel, which exposes a single LevelDB process though HTTP.
But that of course comes at a cost; I need something fast.

Is there any key-value store that:

I only care about reading. Fast multi-process reading is necessary, but not writing.
I'm happy with the current speed of LevelDB, just not with the fact that it is single-process.

Additional details:

Upvotes: 37

Views: 31691

Answers (5)

Amnon
Amnon

Reputation: 2888

There is FaceBook's RocksDB that is supposed to be fast (especially on SSD storage), and there are also others such as LMDB (already mentioned) and WiredTiger

You mentioned Redis - If you'd like to use the Redis API but have one of the above Key/Value databases as the storage instead of your RAM, there are two projects I know of (though haven't tested them): LedisDB (written in Go) and ardb (written in C++).

I've recently started testing what seems like a very promising though yet less known (though I'm sure that will change) key value database library named CuttDB. It has very fast performance and built to handle large amounts of data on the HDD. It even includes a Memcached server interface.

Upvotes: 5

Polor Beer
Polor Beer

Reputation: 2012

You can try ssdb, a redis protocol compatible database built upon leveldb.

https://github.com/ideawu/ssdb

You can use the existing node-redis client, though some of the commands may vary.

benchmarks:

                  Redis (100.000x)
      13,540 op/s ⨠ set small
      13,289 op/s ⨠ set medium
      13,279 op/s ⨠ set large
      13,651 op/s ⨠ get large
      13,681 op/s ⨠ get medium
      14,428 op/s ⨠ get small

                  SSDB (100.000x)
      12,252 op/s ⨠ set small
      11,824 op/s ⨠ set medium
      11,720 op/s ⨠ set large
      13,810 op/s ⨠ get large
      13,593 op/s ⨠ get medium
      12,696 op/s ⨠ get small


                  lmdb (100.000x)
       4,616 op/s ⨠ set small
      11,104 op/s ⨠ set medium
      17,283 op/s ⨠ set large
      13,778 op/s ⨠ get large
      16,002 op/s ⨠ get medium
      50,562 op/s ⨠ get small

                  multilevel (100.000x)
       6,124 op/s ⨠ set small
       5,900 op/s ⨠ set medium
       5,944 op/s ⨠ set large
       6,215 op/s ⨠ get large
       6,125 op/s ⨠ get medium
       6,310 op/s ⨠ get small

As you can see, ssdb is almost as fast as redis, and it is designed for persistent storage. lmdb @didier-spezia mentioned is ultra fast for getting small data, but setting one is slow.

Upvotes: 8

kris.jeong
kris.jeong

Reputation: 99

Why don't you use MySQL(or MariaDB) with Master-slave replication. Based on your requirements. MySql's master-slave architecture is fit for you.

Basically, NoSQL need a lot of server. For example, MongoDB's minimal setting needs three server, HBase needs four server.

In this point of view, If you need more readability then add a new slave server on mysql architecture.

We assume that mysql's read performance is 2k tps. Then four node of mysql's read performance is 8k tps.

It depends on your test result and service usage(read/write ratio).

check below link, that is "Marco Cecconi - The Architecture of StackOverflow". http://www.youtube.com/watch?v=t6kM2EM6so4

Upvotes: 2

Didier Spezia
Didier Spezia

Reputation: 73226

I would suggest to have a look at LMDB (which is the most efficient engine for OpenLDAP, and used in a number of other open-source projects).

LMDB is an embedded key/value store, with a Berkeley-DB or LevelDB like API, does not have to store everything in memory, and can support access from multiple processes. There are Node.js bindings:

Upvotes: 28

AlexGad
AlexGad

Reputation: 6692

The problem you are going to run into is that "lightning fast" and disk don't mix especially if you have random access reads as you do in a key-value system. You need to get as much data into memory as possible since reading from memory is many magnitudes faster than reading from disk.

Is the reason you want to minimize memory because this will be an embedded database? If so, you might want to look at Empress - http://www.empress.com. Have used it in a couple of projects and you can configure how much gets loaded. However, its got the overhead of an RDBMS so not sure it will be as lean as you want.

You might also consider MySQL with the Memcache addon. This allows you to use MySQL as a key value store. Much much faster than regular MySQL since you skip the SQL layer processing. Also, with MySQL, you can turn the knobs to play with how much memory is used.

Firebird is another low memory usage db - http://www.firebirdnews.org/docs/fb2min.html.

Anyway, hope this helps. Without a more indepth explanation of your needs (is this embedded, why the need to save memory and if memory is precious what do you consider low memory consumption, do you need acid, redundancy, what do you consider lightning fast, etc.) its difficult to provide more of an analysis.

Upvotes: 3

Related Questions