CODEWITHSUNDEEP
databasecassandrac10k
jreid42
jreid42

Reputation: 1250

Handling storage of billions of pieces of information?

What is the best solution if I need to have a database with a billion+ objects and I need to have immediate (or nearly immediate) access to any of the items in the database at any time.

This database would be queried at about 1000 requests per second. The rows in the database are pretty much unrelated and thus doesn't need to be relational.

If you're curious why, it's for a simulation of moving elements.

I was thinking something like several load balanced clusters of a Cassandra that are accessed through a load balanced cluster of web servers.

Money is a factor so the cheaper the better. There is no restriction on the software or tool it however must be open source.

Just looking for a database solution that would be good at handling a ridiculous amount of data (does not need to be relational at all) by a large number of users.

It is essential that it handle redundancy and failures.

Just a high level idea to put me in the right direction would be great.

Upvotes: 3

Views: 292

Answers (2)

Andrew
Andrew

Reputation: 3332

One option to consider is mapping your 3D coordinates onto a space-filling curve, effectively representing a point as a single value. Then you could run Cassandra's range queries to get points in an area.

I've seen this implemented in 2D space before, I'm sure it's possible in 3D as well.

Upvotes: 1

Chris
Chris

Reputation: 352

Since you will need to be able to efficiently get all objects within a 3D interval (X_min <= X_obj <= X_max & Y_min <= Y_obj <= Y_max & Z_min <= Z_obj <= Z_max), I am not sure how well a key-value store like Cassandra will suit you. It may be worthwhile to as well have a look at MongoDB since I believe this allows you to index multiple fields and query based on intervals.

Upvotes: 0

Related Questions