CODEWITHSUNDEEP
pythonpython-3.xmultithreadingprogress-barpython-multithreading
garvarma
garvarma

Reputation: 13

Create progress Bar using concurrent.futures in ThreadPoolexecutor process

I have a python script that connects to multiple remote hosts and executes a Linux command to get information. The number of hosts today is around 400 hosts, in that case, I'm using a ThreadPoolExecutor to have all the tasks in the minimum time possible.

All works well and I obtain all data around 100 seconds. The problem is that I don't know what is the state of the process during this period of time, and I would like to add a progress bar that ends when all the Threads finish.

By my side, I added new code to make this progress bar my script using a sleep time, but as I can see in that progress bar is not synchronized with the Thread process (progress bar ends a few seconds before that the Thread process).

Is there a better solution for that? When all this works well, I would like to migrate this progress bar into the Django website.

Here you have part of my code script.

for host in lista_hosts:
        res_versiones[host] = val_defecto
    # print(res_versiones)     
    
LENGTH = len(lista_hosts) # Number of iterations required to fill pbar 
pbar = tqdm(total=LENGTH, desc='consulta_comando') # Init pbar
    
    with ThreadPoolExecutor(200) as executor:    
    
        for host in lista_hosts:
            host_dns = add_dns_cc.add_dns_concesion(host)
         
            res_command_remote = executor.submit(comando_remoto.comando_remoto_hosts, host_dns, comando, res_versiones, user, clave_rsa, passwd)
            
            time.sleep(0.2)
            pbar.update(n=1) # Increments counter
        end = time.time() print(f"Runtime of the program is {end - start}")

Upvotes: 1

Views: 4791

Answers (1)

azelcer
azelcer

Reputation: 1533

Your code updates the progress bar each time a job is submitted to the Executor, but you should update it each time a job is finished.

You can keep the futures returned by submit in a list and check for completion using as_completed:

from concurrent.futures import ThreadPoolExecutor, as_completed

#   ... snip.....

LENGTH = len(lista_hosts)  # Number of iterations required to fill pbar
pbar = tqdm(total=LENGTH, desc='consulta_comando')  # Init pbar
with ThreadPoolExecutor(200) as executor:
    futures = [executor.submit(comando_remoto.comando_remoto_hosts,
                               add_dns_cc.add_dns_concesion(host),
                               comando, res_versiones, user, clave_rsa,
                               passwd) for host in lista_hosts]
    for _ in as_completed(futures):
        pbar.update(n=1)  # Increments counter

Upvotes: 8

Related Questions