CODEWITHSUNDEEP
pythonmultithreadingqueue
kingcope
kingcope

Reputation: 1141

Queue threading status progress

I'm trying to create a web extractor, I have this code for multithreads, and I need print the status/progress of the scanner :

import time
import threading
import Queue       
import sys

try:
    Lista = open(sys.argv[1], "r").readlines()
except(IOError): 
    print "Error: Check your ip list path\n"
    sys.exit(1)

class WorkerThread(threading.Thread) :

    def __init__(self, queue) :
        threading.Thread.__init__(self)
        self.queue = queue

    def run(self) :
        while True :
            counter = self.queue.get()
            sys.stdout.write("line nr : \r")
            self.queue.task_done()      

queue = Queue.Queue()

for i in range(50) :
    worker = WorkerThread(queue)
    worker.setDaemon(True)
    worker.start()

for line in Lista:
    queue.put(line)

queue.join()

print "All task over!"

How can I print the status/progress when the scanner working, I'm tried len(queue) but it doesn't work?

Upvotes: 2

Views: 3719

Answers (1)

abarnert
abarnert

Reputation: 365707

Queue objects do not have a len because, by their very nature, they're being shared across threads, and it would be inaccurate and misleading.

However, they have a qsize method that gives you the approximate size, for exactly this kind of purpose.

If you want exact values, you need a second Queue for that, where each task puts something on the out-queue, and some extra thread (or possibly the main thread) loops over it and counts up the tasks done so far. Or, alternatively, something simpler, like a global int counter with a global Lock protecting it.

However, I think it would be a lot simpler to write this in terms of a pool or executor. That would take care of queueing up tasks for you, and returning a value for each one to the main thread, without you needing to manage anything. For example, using futures, the backport of the Python 3.x concurrent.futures module for 2.x, here's your whole program, with progress added:

import sys
import futures 

try:
    Lista = open(sys.argv[1], "r").readlines()
except(IOError): 
    print "Error: Check your ip list path\n"
    sys.exit(1)

def task(line):
    # Do something

with futures.ThreadPoolExecutor(50) as executor:
    fs = [executor.submit(task, line) for line in Lista]
    for i, f in enumerate(futures.as_completed(fs)):
        sys.stdout.write("line nr: {} / {} \r".format(i, len(Lista)))

print "All task over!"

Upvotes: 3

Related Questions