CODEWITHSUNDEEP
pythonmultiprocessingpermutationproductpython-itertools
30114
30114

Reputation: 159

Python itertools multiprocessing

I'm creating huge array of permutations with replacement (product) and it demands so much time for computation. Let's use simple function like this one:

def permutations(li):
    return [p for p in itertools.product(li, repeat=20)]

permutation(range(11))

I'm simply not sure how to split this function into multiple processes (using "Pool" class or similar). I have never used multiprocessing in Python and I'd like to ask for help.

Any hints?

Thanks!

Upvotes: 2

Views: 2314

Answers (1)

chapelo
chapelo

Reputation: 2562

Too big divided by 4 is still Too big. Heed unutbu's comment: It would take forever divided by 4 doing it with multiprocessing.

However, here is a working example of just 3**2, assuming that you will do something useful in your_process:

import itertools
import multiprocessing

def your_process(perm):
    # this is where you process each permutation
    # currently it just prints the permutation.
    print(perm)

def permutations(li):
    with multiprocessing.Pool(4) as workers:
        workers.map(your_process, itertools.product(li, repeat=2))

permutations(range(3))

But if you really have >11**>20 permutations, consider using something like distributed computing or BOINC, or perhaps re-think your algorithm.

Upvotes: 5

Related Questions