CODEWITHSUNDEEP
pythonmemorypickle
tevang
tevang

Reputation: 518

How to load only selected objects from a pickle file?

I have multiple pickle files which contain two lists. The first one is a lightweight list of floats, but the second one is a list of long vectors which consumes a lot of memory when loaded. How can I load only the first list from each pickle file? This is the way I currently do it:

import cPickle, bz2    
with bz2.BZ2File(lhl_file, "rb") as pickled_file:
    pickle_object = cPickle.load(pickled_file)
    light_list, \
    heavy_list = list(pickle_object)

Upvotes: 2

Views: 2884

Answers (1)

F.S.
F.S.

Reputation: 1261

If you pickled two lists individually into a pickle file, you should be able to load them individually easily.

Here is an example I adapted from another answer, which covers more thoroughly.

import pickle

filename = "pickle.dat"
data1 = 'hello'
data2 = 'world'
data3 = 'buddy'

# pickle one object at at time
f1 = open(filename, "wb")
pickle.dump(data1, f1)
pickle.dump(data2, f1)
pickle.dump(data3, f1)
f1.close()

# so that you can load them one at at time
f2 = open(filename, 'rb')
print pickle.load(f2)
print pickle.load(f2)
f2.close()

Output: 

hello
world

PS: As far as I know, it's not possible to load data2 before data1. Though that is not the case for you it seems.

Upvotes: 2

Related Questions