CODEWITHSUNDEEP
pythondatabaseredisnosql
Johannes Pertl
Johannes Pertl

Reputation: 983

Which database for a huge list of one type of data?

A python script I write generates IDs that I need to add to a list, while another script checks, if the ID already exists in the list. There are no other tables, relations or anything, just a huge, growing list of 6-letter strings. I need to add and read values as fast as possible at the same time. What would be the database of choice in that case? A NoSQL database like redis?

Upvotes: 0

Views: 179

Answers (2)

tabreaz
tabreaz

Reputation: 659

Redis is a perfect fit for this, you can use redis hashes. Also ensure you shard hashes for efficiency see below link for more details on how instagram reduces the total memory requirements by using redis hashes.

https://instagram-engineering.com/storing-hundreds-of-millions-of-simple-key-value-pairs-in-redis-1091ae80f74c

Upvotes: 1

user5919789
user5919789

Reputation: 45

Yes, MongoDB or Redis, can do the work

Upvotes: 0

Related Questions