CODEWITHSUNDEEP
sqlmongodbstorage
James
James

Reputation: 605

Large data store (nosql or not)

I have large amounts of scientific data that I need to store (150 TB+ starting data) and I want to know the best way to store the data (nosql or RDBMS etc...)

Any tips......

James

Upvotes: 0

Views: 575

Answers (5)

Jim Hunziker
Jim Hunziker

Reputation: 15380

You should look at NetCDF and HDF5. Also, consider using PyTables for accessing and extracting the data.

Upvotes: 0

Enrico
Enrico

Reputation: 2037

This really depends on what you need to do with the data on a later time. If the data is a collection of a few very large files then the a normal file system would be ok. If you need to be able to search and analyse the data then a database might be the best solution.

I am working with large datasets as well in a scientific environment. Most of this data is tabular and when we started we stored every datapoint is a table. We found it to be much easier in the end to zip the tables and store this in a binary blob into the database. In a separate table we stored the metadata about this tables.

Upvotes: 1

Warren
Warren

Reputation: 101

Does it have to be one database type? Part of NoSQL means one size does not fit all, so why not two or more NoSQL? How about one column store and one graph database?

Upvotes: 0

TTT
TTT

Reputation: 2375

There are special db's for scientific data: http://www.dbms2.com/2009/09/12/xldb-scid/

Upvotes: 0

rano
rano

Reputation: 5666

Answer this question to choose from NoSQL or a RDBMS : "Are my data structured in relationships?"

Upvotes: 2

Related Questions