CODEWITHSUNDEEP
pythonmemory-managementout-of-memoryshared-memorydynamic-memory-allocation
yptheangel
yptheangel

Reputation: 112

How to replace large python dictionary with a memory efficient data structure?

I use a python dictionary to store key value pairs and the dictionary gets too large(>100GB) and hits a memory limit.

What is a better memory efficient data structure to store key value pairs in python? E.g. we can use generators to replace lists

Upvotes: 1

Views: 516

Answers (2)

dagnelies
dagnelies

Reputation: 5329

Maybe that could help: https://github.com/dagnelies/pysos

It keeps only the index in memory and keeps the data on disk.

Upvotes: 0

Taron Qalashyan
Taron Qalashyan

Reputation: 723

You can use sqlitedict which provides key-value interface to SQLite database. About memory usage. SQLite doesn't need your dataset to fit in RAM. By default it caches up to cache_size pages, which is barely 2MB .

Upvotes: 2

Related Questions