CODEWITHSUNDEEP
pythonqueueblockingqueue
Ciasto piekarz
Ciasto piekarz

Reputation: 8277

why queue is showing incorrect data?

I have used queue for passing urls to download, however the queue gets corrupted when received in the thread:

class ThreadedFetch(threading.Thread):
    """ docstring for ThreadedFetch
    """
    def __init__(self, queue, out_queue):
        super(ThreadedFetch, self).__init__()
        self.queue = queue
        self.outQueue = out_queue

    def run(self):
        items = self.queue.get()
        print items

def main():
    for i in xrange(len(args.urls)):
        t = ThreadedFetch(queue, out_queue)
        t.daemon = True
        t.start()

    # populate queue with data 
    for url, saveTo in urls_saveTo.iteritems():
        queue.put([url, saveTo, split])
    # wait on the queue until everything has been processed
    queue.join()

output resulting execution of run() when I execute the main is :

['http://www.nasa.gov/images/content/607800main_kepler1200_1600-1200.jpg', ['http://broadcast.lds.org/churchmusic/MP3/1/2/nowords/271.mp3', None, 3None, 3]
]

while expected is 

['http://www.nasa.gov/images/content/607800main_kepler1200_1600-1200.jpg', None, 3]
['http://broadcast.lds.org/churchmusic/MP3/1/2/nowords/271.mp3', None, 3]

Upvotes: 0

Views: 161

Answers (2)

hughdbrown
hughdbrown

Reputation: 49053

Populate your queue before you start your threads. Add a lock for I/O (for the reason @tdelaney says -- the threads are interleaving writes to stdout and the results appear broken). And modify your run method to this:

lock = threading.Lock()

def run(self):
    while True:
        try:
            items = self.queue.get_nowait()
            with lock:
               print items
        except Queue.Empty:
            break
        except Exception as err:
            pass
        self.queue.task_done()

You might also find that it is easier to do this with concurrent.futures. There is a solid example of using a method that returns a value that is called in a thread pool.

Upvotes: 0

tdelaney
tdelaney

Reputation: 77407

All of the threads print their data at once and the results are interleaved. If you want threads to display data in production code, you need some way for them to cooperate when writing. One option is a global lock that all screen writers use, another is the logging module.

Upvotes: 1

Related Questions