CODEWITHSUNDEEP
pythonrandomparallel-processingjoblib
Ruoxu
Ruoxu

Reputation: 21

Random number generator using Joblib

I need to generate random number in a function which is paralleled using Joblib. However, the random number generated from the cores are exactly the same.

Currently I solved the problem by assigning random seeds for different cores. Is there any simple way to solve this problem?

Upvotes: 2

Views: 957

Answers (1)

Gael Varoquaux
Gael Varoquaux

Reputation: 2476

This is expected, although unfortunate.

The reason is that joblib (based on the standard multiprocessing Python tool) relies on forking under Unix. Forking creates the exact same processes and thus the same pseudo-random number generation.

The right way to solve this problem is to pass to the function that you are calling in parallel a seed for each call, eg a randomly-generated integer. That seed is then used inside the function to seed the local random number generation.

Upvotes: 3

Related Questions