CODEWITHSUNDEEP
pythonmemoryvirtual-memory
Mkoch
Mkoch

Reputation: 2014

handling huge arrays in python

My program runs a simulation which requires huge objects to store the data in. The size of the blob is larger than 2-3GB. Even though I should have anough memory in my MBP, python (Python 2.7.3 on Mac OS X, from ports) cannot seem to use it all, and the system gets totally frozen.

To save the status of the simulation, I use pickle, but it also doesn't work for too large objects, it seems as if pickle would duplicate the objects in the memory before dumping them...

QUESTION: is there a standard library which can handle huge python data structures (dict, set, list) without keeping them in the memory all the time? Alternatively is there a way to force python to run in virtual memory? (I'm not very familiar with numpy, would it help me in this situation?)

Thanks in advance!

Upvotes: 2

Views: 1292

Answers (1)

Fabian
Fabian

Reputation: 4348

If you are using the 64bit version of Python and still run into problems with pickle or other built-in modules, you can store the Python objects in an object-orientated database instead. We're working with large objects (~10GB) here everyday and use ZODB for that. It's not the fastest but gets the job done.

I also hear that dobbin might be a good alternative.

Upvotes: 2

Related Questions