CODEWITHSUNDEEP
databaseperformanceblobstorage
Gourab Roy
Gourab Roy

Reputation: 11

Options for storing binary data of size 100 MB to 1 GB

If I am planning to store binary data of size varying from 100 MB to 1 GB what is a better option 1. Relational data (this I am assuming would be a direct no) 2. NoSQL DB 3. File system

Also do suggest if anything else would be better (in terms of performance to write and read the data)

Nature of data: It's a table having 1-50 Million cells. There is not much structural relationship between the cells. They can be considered independent. The table is never updated. It's an insert only table. The above information tells me (though limited by lack of knowledge about NoSQL DBs) that a columnar DB won't be much of help. Do correct me.

Thanks

Upvotes: 1

Views: 679

Answers (1)

root
root

Reputation: 6048

tl;dr: Filesystem.

Since there are no relationships between the items being stored, a relational DB would be a huge waste of all possible resources.

Likewise for any NoSQL DB which is fancier than a key-value store e.g. Graph Databases. Even something like Cassandra would be overkill.

So the choice is generally between some a key-value store, object storage e.g. Minio, or regular POSIX filesystem.

All 3 have both simple implementations suitable for testing and complex implementation with replication etc. suitable for production.

Of those, I would recommend using a filesystem.

A filesystem would be more suitable than a key-value store because the data is big, and key-value stores are generally optimized for quantity and latnecy, rather than size and bandwidth.

A filesystem would also be more useful than an object store or a key-value store because most filesystems have the same API (POSIX), which means that:

  1. It's easy to just switch to another filesystem. There are many, many
  2. tools to do just about anything with data stored on a filesystem.

Upvotes: 1

Related Questions