CODEWITHSUNDEEP
pythonmultiprocessing
kumaji
kumaji

Reputation: 43

python ProcessPoolExecutor do not work when in function

python ProcessPoolExecutor works in command lines but not running after adding to a function

it is working like this

from concurrent import futures

def multi_process(func, paras, threads):
    with futures.ProcessPoolExecutor(max_workers=threads) as pool:
        res = pool.map(func, paras, chunksize=threads)
    return list(res)
p = multi_process(func,paras,threads)

but not working at all as below

def upper(paras,threads):
    def func:
        some func
    def multi_process(func, paras, threads):
        with futures.ProcessPoolExecutor(max_workers=threads) as pool:
            res = pool.map(func, paras, chunksize=threads)
        return list(res)
    p = multi_process(func,paras,threads)
    return p
p = upper(paras,threads)

no warning or error but without any response for a long time.

Upvotes: 4

Views: 7276

Answers (1)

neotrinity
neotrinity

Reputation: 230

Your do get an error. Its.

AttributeError: Can't pickle local object 'upper.<locals>.func'.

The reason is for multiprocessing to work it needs the function to be defined at the global level.

To achieve what you want you can do the following:

from concurrent import futures

# Has to be a global function
def func(para):
    print(para)


def upper(paras,threads):
    # This cannot be a local function.
    #def func(para):
    #    print(para)
    def multi_process(func, paras, threads):
        with futures.ProcessPoolExecutor(max_workers=threads) as pool:
            res = pool.map(func, paras, chunksize=threads)
        return list(res)
    p = multi_process(func, paras, threads)
    return p

paras = [1, 2, 3]
threads = 3
p = upper(paras,threads)

Upvotes: 6

Related Questions