CODEWITHSUNDEEP
parallel-processinggoogle-cloud-mljoblib
Pablo
Pablo

Reputation: 3514

running joblib.Parallel(mlxtend) does not scale in cloud-ml

Im running a job using the mlxtend library. Specifically the sequential_feature_selector that is parallelized using joblib.Parallel source. When I run the package on my local computer it uses all the available CPUs, but when i send the job to cloud-ml it only uses one core. It doesn't matter what is the number that i put in the n_jobs parameter. I´ve also tried with differents machine types but same thing happen. Does anybody know what the problem might be ?

Upvotes: 0

Views: 282

Answers (1)

Pablo
Pablo

Reputation: 3514

For anyone that might be interested, we solve the problem fixing the sklearn version in the setup.py to the 0.20.2. we had sklearn in the packages before, but without a version. 

#setup.py
from setuptools import find_packages
from setuptools import setup

REQUIRED_PACKAGES = ['joblib==0.13.0',
                     'scikit-learn==0.20.2',
                     'mlxtend']

Upvotes: 1

Related Questions