I need to choose a database for storing statistical data (in fact this is a series of timestamp-value data). I understand that virtually any database can handle this, but there are a couple of requirements: it should be fast; it should be able to handle A LOT of data (10s of gigabytes) and splice it fast; it should have a stable, maintained and handy interface to Erlang; it should be available from Python; it should be able to make something like the thing named "capped collections" in mongodb: collection with the capped size, with old data being rewritten if the size reach the limit. I thought about mongo, but emongo seems to be a little dead - the last commit was made 7 months ago.

Reputation: 141

Best database for storing statistics from Erlang

I need to choose a database for storing statistical data (in fact this is a series of timestamp-value data). I understand that virtually any database can handle this, but there are a couple of requirements:

it should be fast;
it should be able to handle A LOT of data (10s of gigabytes) and splice it fast;
it should have a stable, maintained and handy interface to Erlang;
it should be available from Python;
it should be able to make something like the thing named "capped collections" in mongodb: collection with the capped size, with old data being rewritten if the size reach the limit.

I thought about mongo, but emongo seems to be a little dead - the last commit was made 7 months ago.

Upvotes: 5

Answers (4)

Jeremy Raymond

Reputation: 6027

Riak may be a good choice (here's a Riak comparison to MongoDB). It's written in Erlang, is distributed, fault tolerant and scales linearly. It has clients for Erlang, Javascript, Java, PHP, Python, Ruby. A REST interface, a protobuf interface and so many other goodies (Map Reduce, links, replication, pre/post commit hooks, ...). It's open source and is created maintained by Basho. Basho has commercial offering of Riak as well with some extra features (like multi-site replication, SNMP monitoring, etc) but there's awsome value in the OS version.

Depending on your needs it may make sense to combine a couple of technologies. For example you could front your system with an in memory store like Redis for speed and use Riak to persist the data. Redis + Riak is a pretty sweet stack.

Upvotes: 3

Eric

Reputation: 2706

redis is quite a close contender.

The only current limitation is the size of the dataset, which has to be either store in full in memory or use the VM method, in which only the key space has to fit in memory (however a bit of spare room for actual data would be nice) but has a very slow startup time.

Antirez, the developer, is rewriting the backend into something called diskstore which should solve your issue. It's not baked yet, but I have a lot of confidence in this project.

About the capped collections, redis does not have a direct way for handling that. But the LTRIM function can help you out.

Upvotes: 0

I GIVE CRAP ANSWERS

Reputation: 18879

Files on disk, rotated, will serve your demands fine. The point is you don't want to search data quickly.

Upvotes: 1

0xAX

Reputation: 21817

I think postgresql and pgsql driver it will be best solution for you.

Upvotes: 1

Best database for storing statistics from Erlang

Answers (4)

Related Questions