CODEWITHSUNDEEP
pythonmultiprocessing
alberto
alberto

Reputation: 2645

multiprocessing: a progress bar for each task

I have a class that implements an iterative algorithm. Since the execution takes a while and I need to average the results over many executions, I decided to use multiprocessing. The thing is that I would like to have a progress bar (or something less fancy) for every execution. Something like:

experiment 1 [##########] 60%
experiment 2 [#############] 70%
experiment 3 [###] 20%

My class looks like this (note that I already used a progress bar and that I would like to keep it there so that I continues to work when I do not parallelize):

from __future__ import division
from time import sleep

class Algo():

    def __init__(self, total_iters, arg1, arg2, name):
        self.total_iters = total_iters
        self.arg1 = arg1
        self.arg2 = arg2
        self.name = name

    def step(self, iteration):
            """
            One iteration of Algorithm
            """
            # Progress bar
            completed = 100*iteration/self.total_iters
            if completed == 0: print ""
            print '\r {2} [{0}] {1}%'.format('#'*(int(completed/5)), completed, self.name),

            # Do some stuff
            sleep(0.001)

    def run(self):
        for i in xrange(self.total_iters):
            self.step(i)

        #  Output the final result in unique file

And this is my try:

import multiprocessing as mp 

if __name__ == "__main__":

    algo1 = Algo(200, 0,0, "test1")
    pool = mp.Pool(processes=3)
    for i in xrange(3):
        pool.apply_async(algo1.run) # in real life run will be passed N arguments

    pool.close()
    pool.join()

Any ideas?

PS: I am trying to avoid curses

Upvotes: 2

Views: 1772

Answers (1)

nullptr
nullptr

Reputation: 660

Quick and dirty and in python 3, but you will get the idea ;)

import random
import time
import multiprocessing
import os
import collections


class Algo(multiprocessing.Process):
    def __init__(self, steps, name, status_queue):
        multiprocessing.Process.__init__(self)
        self.steps = steps
        self.name = name
        self.status_queue = status_queue

    def step(self, step):
            # Progress bar
            self.status_queue.put((self.name, (step+1.0)/self.steps))
            # Do some stuff
            time.sleep(0.1)

    def run(self):
        for i in range(self.steps):
            self.step(i)


def print_progress(progress):
    # Windows:
    os.system('cls') 
    for name, percent in progress.items():
        percent = int(percent * 100)
        bar = ('#' * int(percent/10)) + (' ' * (10 - int(percent/10)))
        print("{}: [{}] {}%".format(name, bar, percent))

if __name__ == "__main__":
    status = multiprocessing.Queue()
    progress = collections.OrderedDict()
    algos = [Algo(random.randrange(100, 200), "test" + str(i), status) for i in range(3)]

    for a in algos:
        a.start()

    while any([a.is_alive() for a in algos]):
        while not status.empty():
            name, percent = status.get()
            progress[name] = percent
            print_progress(progress)

        time.sleep(0.1)

Upvotes: 3

Related Questions