CODEWITHSUNDEEP
pythonpickle
user2129623
user2129623

Reputation: 2257

pickling defaultdict with lambda

I am trying to pickle defaultdict(), but couldn't do. What is the correct way?

pos = defaultdict(lambda: 0)
neg = defaultdict(lambda: 0)    

countdata = self.getCountdata(pos, neg, totals)
cPickle.dump(countdata, open(CDATA_FILE, 'w'))

This gives:

Traceback (most recent call last):
  File "sentiment_worker.py", line 146, in <module>
    MyDict().gearman_worker.work()
  File "sentiment_worker.py", line 28, in __init__
    self.train()
  File "sentiment_worker.py", line 91, in train
    cPickle.dump(countdata, open(CDATA_FILE, 'w'))
  File "/usr/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle function objects

When I tried dill:

import dill
dill.dumps(countdata, open(CDATA_FILE, 'w') )

which gives:

 File "sentiment_worker.py", line 151, in <module>
    if __name__ == '__main__':
  File "sentiment_worker.py", line 29, in __init__
    self.train()
  File "sentiment_worker.py", line 96, in train
    # cPickle.dump(countdata, open(CDATA_FILE, 'w'))
  File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 243, in dumps
    dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
  File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 204, in dump
    pik = Pickler(file, protocol)
  File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 387, in __init__
    StockPickler.__init__(self, *args, **kwds)
  File "/usr/lib/python2.7/pickle.py", line 202, in __init__
    raise ValueError("pickle protocol must be <= %d" % HIGHEST_PROTOCOL)
ValueError: pickle protocol must be <= 2

What is the correct way to pickle and unpickle the content?

Upvotes: 3

Views: 1533

Answers (1)

Mike McKerns
Mike McKerns

Reputation: 35217

You can do it with dill. You've made a typo… you should have used dill.dump instead of dill.dumps if you want to dump to a file. If you want to dump to a string, use dumps.

>>> import dill
>>> from collections import defaultdict
>>> pos = defaultdict(lambda: 0)
>>> neg = defaultdict(lambda: 0)
>>> countdata = (pos,neg)
>>> _countdata = dill.loads(dill.dumps(countdata))
>>> _countdata
(defaultdict(<function <lambda> at 0x10917f7d0>, {}), defaultdict(<function <lambda> at 0x10917f8c0>, {}))
>>>
>>> # now dump countdata to a file 
>>> with open('data.pkl', 'wb') as f:
...     dill.dump(countdata, f)
...
>>>

Upvotes: 2

Related Questions