CODEWITHSUNDEEP
pythonpython-3.xmultiprocessingtqdm
Sraw
Sraw

Reputation: 20224

How to use tqdm through multi process in python?

I'm trying to use tqdm through multi processes. And the behavior is not as expected. I think the point is that the value of pbar doesn't update through the processes. So how to deal with this problem? I have also tried to use Value to update pbar.n manually, but still failed. It seems tqdm doesn't support update value and render manually.

def test(lock, pbar):
    for i in range(10000):
        sleep(0.1)
        lock.acquire()
        pbar.update()
        lock.release()

pbar = tqdm(total = 10000)
lock = Lock()
for i in range(5):
    Process(target = test, args = (lock, pbar))

Upvotes: 11

Views: 9262

Answers (1)

unutbu
unutbu

Reputation: 880189

Generally, each process has its own data, independent of every other process. Spawning a new process (which calls os.fork on Unix) creates a copy of the current process. Each process obtains its own copy of all global values (such as pbar). Each process's global variables may share the same names as variables in the other processes, but each can hold an independent value.

In your case it looks like you want just one pbar to exist, and all calls to update should update that one pbar. So create pbar in only one process, and use a Queue to send signals to that process to update pbar:

import multiprocessing as mp

SENTINEL = 1

def test(q):
    for i in range(10000):
        sleep(0.1)
        q.put(SENTINEL)

def listener(q):
    pbar = tqdm(total = 10000)
    for item in iter(q.get, None):
        pbar.update()

if __name__ == '__main__':
    q = mp.Queue()
    proc = mp.Process(target=listener, args=(q,))
    proc.start()
    workers = [mp.Process(target=test, args=(q,)) for i in range(5)]
    for worker in workers:
        worker.start()
    for worker in workers:
        worker.join()
    q.put(None)
    proc.join()

Upvotes: 15

Related Questions