CODEWITHSUNDEEP
pythonobjectpandasdataframestorage
drogers
drogers

Reputation: 11

Pandas storing 1000's of dataframe objects

I am working on a large project that does SPC analysis and have 1000's of different unrelated dataframe objects. Does anyone know of a module for storing objects in memory? I could use a python dictionary but would like it more elaborate and functional mechanisms like locking, thread safe, who has it and a waiting list etc? I was thinking of creating something that behaves like my local public library system. The way it checks in and out books to one owner ...etc.

Upvotes: 1

Views: 1736

Answers (2)

Jeff
Jeff

Reputation: 129018

HDF5-pytables is a pretty good storage back-end for pandas.

Pytables can expand to be managed like a file system BUT care must be taken if you need any kind of multi-access (if you really need that you should use a traditional database that provides these features).

Upvotes: 2

reptilicus
reptilicus

Reputation: 10397

Redis with redis-py is one solution. Redis is really fast and there are nice Python bindings. Pytables, as mentioned above, is a good choice as well. PyTables is HDF5, and is really really fast.

Upvotes: 1

Related Questions