CODEWITHSUNDEEP
pythonpython-2.7pickle
Joel Green
Joel Green

Reputation: 920

Pickle Queue objects in python

I have a class that uses a list of Queue objects. I need to pickle this class including the information saved in the queue objects. For example:

import Queue
import pickle

class QueueTest(object):
    def __init__(self):
        self.queueList = []
    def addQueue(self):
        q = Queue.Queue()
        q.put('test')
        self.queueList.append(q)


obj = QueueTest()
obj.addQueue()

with open('pickelTest.dat','w') as outf:
    pickle.dump(obj,outf)

returns the error

raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle lock objects

Is there a work around to pickle Queue objects?

Upvotes: 10

Views: 8770

Answers (2)

Blckknght
Blckknght

Reputation: 104722

I suggest replacing your uses of Queue.Queue with collections.deque. The Queue class is intended to be used for synchronized communication between threads, so it will have some unnecessary overhead when used as a regular data structure. collections.deque is a faster alternative. (The name "deque" is pronounced "deck" and means "double-ended queue".)

The deque class does have a different API than the Queue type, but it should be pretty easy to translate between them. Use deque.append in place of Queue.put and deque.popleft in place of q.get() (or appendleft and pop, if you feel like going the other direction). Rather than calling Queue.empty, just use a deque instance as a Boolean value (like you do to test for an empty list).

deque instances are picklable:

>>> import collections, pickle
>>> q = collections.deque(["test"])
>>> pickle.dumps(q)
b'\x80\x03ccollections\ndeque\nq\x00]q\x01X\x04\x00\x00\x00testq\x02a\x85q\x03Rq\x04.'

Upvotes: 14

jamylak
jamylak

Reputation: 133574

As you have commented to @Blckknght, you don't need the synchronization features of Queue.Queue. So just use collections.deque that the Queue.Queue class uses itself as the underlying queue data structure. You will have to use .appendleft to emulate the FIFO Queue.put and .pop to emulate Queue.get

Upvotes: 4

Related Questions