CODEWITHSUNDEEP
pythonmultithreadingqueuepool
Tyler
Tyler

Reputation: 39

Passing Queue to Threadpool - Python

Getting confused using pool with list vs queues. I get the error: TypeError: 'int' object is not iterable when passing q.get in place of a list. Hopefully it's just an oversight.

from multiprocessing.dummy import Pool 
import Queue

numbers = [11,12,13,14]

def printNumbers(n):
    print n

q = Queue.Queue()    
for i in (1,2,3,4):
   q.put(i)

#multiThread with list numbers
pool = Pool(4) 
results = pool.map(printNumbers, numbers)
pool.close() 
pool.join() 

#multiThread with queue
pool2 = Pool(4) 
results2 = pool2.map(printNumbers, q.get())
pool2.close() 
pool2.join()

Upvotes: 1

Views: 1819

Answers (1)

foundart
foundart

Reputation: 370

The Queue class does not implement the iterable protocol. You'll need to provide your own Queue implementation that does. Here is an example: 

class IterableQueue(Queue.Queue):

    _sentinel = object()

    def __init__(self, maxsize=0):
        Queue.Queue.__init__(self, maxsize)
        self.put(self._sentinel)

    def __iter__(self):
        return iter(self.get, self._sentinel)

You can read the details of __iter__ and how the sentinel is used at https://docs.python.org/2/library/functions.html#iter

Please note that my example queue is not what you want if you want a queue that will block when empty until more items are added. For that, see Python iterable Queue

However, when I used that version with your code, it never terminated. To use that one, you'll need to restructure a bit so that you can call the close method when you know you have no more items to add to the queue.

Upvotes: 2

Related Questions