CODEWITHSUNDEEP
pythonmemory-managementpickle
aceron
aceron

Reputation: 3

Python - Using cPickle to load a previously saved pickle uses too much memory?

Python - Using cPickle to load a previously saved pickle uses too much memory?

My pickle file is about 340MB but takes up 29% of 6gb of memory when loaded. This seems a bit too much. The pickle file is a dictionary of dictionaries. Is this appropriate?
Code used:

import cPickle as pickle

file = pickle.load( file_handle )

Thanks

Upvotes: 0

Views: 849

Answers (2)

optixx
optixx

Reputation: 2120

I always had memory problems with big pickels and sub dicts. So i ended up writing my objects via pprint into files and later i import that files via a a custom module loader to get the data back in the process scope. Works fine and doesn't waste memory.

Upvotes: 1

sth
sth

Reputation: 229673

About 1.7GB seems a bit much, but not impossible. How much memory did the data take before it was pickled?

After unpickling the data should take about the same amount of memory as it took before it was pickled, how big it is in it's on-disk format is not really that significant.

Upvotes: 0

Related Questions