CODEWITHSUNDEEP
pythonmultiprocessingshared-memory
aroville
aroville

Reputation: 63

Multiprocessing: append to 2 lists simultanously

I have this code:

from multiprocessing import Pool, Manager
import numpy as np

l = Manager().list()

def f(args):
    a, b = args
    l.append((a, b))


data = [(1,2), (3,4), (5,6)]
with Pool() as p:
    p.map(f, data)
x, y = np.transpose(l)

# do something with x and y...

In reality, data is an array with lots of values, and the transpose operation is long and memory consuming.

What I would like is to append "a" and "b" directly to lists x and y to avoid the transpose operation. It is important that the output keeps the correspondence in data and looks like so: [[1,3,5], [2,4,6]]

What would be a smart way to do that?

Upvotes: 5

Views: 1332

Answers (1)

falsetru
falsetru

Reputation: 369134

Instead of trying to append from sub-processes, you can make the function return the values and append them in main process; you don't need to care about mutual access between sub-processes (also no need to use manager).

from multiprocessing import Pool


def f(args):
    a, b = args
    # do something with a and b
    return a, b


if __name__ == '__main__':
    data = [(1,2), (3,4), (5,6)]
    x, y = [], []
    with Pool() as p:
        for a, b in p.map(f, data):   # or   imap()
            x.append(a)
            y.append(b)

    # do something with x and y
    assert x == [1,3,5]
    assert y == [2,4,6]

Upvotes: 7

Related Questions