CODEWITHSUNDEEP
pythonmultiprocessingtqdm
roundtheworld
roundtheworld

Reputation: 2795

Using tqdm with multiprocessing

I am using multiprocessing in Python 3.7 to kickoff several long running operations. Each individual process takes a couple minutes to complete. I want a progress bar for each individual process that is running displayed simultaneously. I've been having trouble getting this to work as the progress bars are constantly jumping around and replacing each other in my current futile attempt.

I create my processes like this:

with Pool(processes=cpu_count()) as pool:
    return pool.starmap(_execute_long_running_process, items)

Inside my _execute_long_running_process function is a for loop that I want the progress bar to track.

for i in tqdm(list_of_things_to_process, leave=False):
    # do a thing

I believe my issue arises from each tqdm progress bar existing in its own process separate from the main process. I'm not sure what the workaround would be to make them behave as I expect where I have several progress bars each being updated.

Upvotes: 5

Views: 7391

Answers (2)

NRVulture
NRVulture

Reputation: 30

If you are using PyCharm then enabling Emulate terminal in output console inside the run/debug configurations might help. At least it works for me. This solution was found on this thread after searching for hours.

To clarify, I ran into the same "progress bar jumps around" issue as you except I was using multi-threading. In each thread there is a loop that runs for quite a while, and I want to display the progress of each loop separately.

This issue is in their FAQ (under "Nested progress bars"). It just happens that the way I triggered it is not listed there.

Upvotes: 0

J_H
J_H

Reputation: 20495

I obtained good results by structuring my computation in this way:

    with multiprocessing.Pool() as p:
        list(tqdm.tqdm(p.imap_unordered(inserter, tasks), total=len(tasks)))

That gives you just a single, unified progress bar for all tasks.

Upvotes: 5

Related Questions