CODEWITHSUNDEEP
pythonfor-loopqueuemultiprocessing
3quanfeng
3quanfeng

Reputation: 39

'JoinableQueue' object is not iterable

Assuming i have a queue: 

myqueue = multiporcessing.JoinableQueue()

In my code, the parent process puts all the object (e.g. a string) into myqueue, 'myqueue' will be shared by child processes, and the child process will need to check if an object (e.g. xxx) is in myqueue, I tried to use: 

if xxx in [y for y in myqueue]

but it gives me the error like:

'JoinableQueue' object is not iterable

My question is, is there any other way I can use to get what I want as above? Or any way I can use to check if an object is still in the queue?

The reason for checking the shared queue is that the objects in the queue is interdependence, and every object has some status such as 'running', 'waiting', 'completed'. For example, object 'a' has a dependency 'A', 'A' and 'a' will be put into the queue by parent process(before child process kick-off), when a child process gets 'a' from the queue, it needs to check the status of 'A', if the status of 'A' is still 'waiting', then 'a' will be put back into the queue unless 'A' is 'completed'. In other words, 'A' is not in myqueue anymore.

Upvotes: 0

Views: 1273

Answers (1)

otus
otus

Reputation: 5732

JoinableQueue doesn't support peeking of any kind (unless you count empty/full).

Depending on what you want to do with the collection in addition, you could need a different object to store your queue in. However, there are two possible solutions:

  1. If you know no other process will touch the queue while you are checking this, you could get every object, see if what you were looking for was there, then put them back in the same order.

  2. Otherwise, if the child is the only process that will ever get objects, you could put the objects in another temporary queue.

The former should be straightforward to implement. The latter could be something like:

class QueueView:
    def __init__(self, joinable):
        self.joinable = joinable
        self.other = collections.deque()
    def get(self, block=True, timeout=None):
        if self.other:
            return self.other.popleft()
        self.joinable.get(block, timeout)
    def __contains__(self, item):
        if item in self.other:
            return True
        try:
            while True:
                o = self.joinable.get(block=False)
                self.other.append(o)
        except Queue.Empty:
            pass
        return item in self.other

Only one process should see the QueueView object and get anything from the JoinableQueue through the view object. Any number of processes can put things in the JoinableQueue.

Upvotes: 1

Related Questions