CODEWITHSUNDEEP
pythonscikit-learnparallel-processingjoblib
Leockl
Leockl

Reputation: 2156

When specifying say n_jobs=-1 in scikit-learn's packages, do we need to first import joblib.Parallel?

When using scikit-learn's packages which has the option to choose a value for n_jobs for parallel processing, do we need to first import joblib.Parallel or will the scikit-learn package work with parallel processing without needing to first import joblib.Parallel.

Some of scikit-learn packages which has parallel processing are:

etc.

Upvotes: 0

Views: 352

Answers (1)

user3666197
user3666197

Reputation: 1

Q : "... do we need to first import joblib.Parallel or will the scikit-learn package work with parallel processing without needing to first import joblib.Parallel." ?

The B is correct : scikit-learn will work, as it was properly designed and implemented, so as to manage its own internal needs to import whatever package it knowingly depends on. This is a professional software standard to take due care of its own internal dependencies, isn't it?

BONUS PART for those indeed interested in understanding WHY :

Those who indeed want to see the trick - check the way of inheritance done on one of the packages source-level, obvious from the files reported below :

(base) Tue Mar 17 12:00:34 a64FX:~$ grep -R "joblib.Parallel" /home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/
            /home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/decomposition/online_lda.py:        parallel : joblib.Parallel (optional)
            /home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/decomposition/online_lda.py:            Pre-initialized instance of joblib.Parallel.
            /home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/decomposition/online_lda.py:        parallel : joblib.Parallel
            /home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/decomposition/online_lda.py:            Pre-initialized instance of joblib.Parallel
Binary file /home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/decomposition/__pycache__/online_lda.cpython-35.pyc matches
            /home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/externals/joblib/parallel.py:                print("Using %s as joblib.Parallel backend instead of %s "
            /home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/externals/joblib/parallel.py:    """Callback used by joblib.Parallel's multiprocessing backend.
            /home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/externals/joblib/_dask.py:        joblib.Parallel will never access those results
            /home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/externals/joblib/_dask.py:        # See 'joblib.Parallel.__call__' and 'joblib.Parallel.retrieve' for how
Binary file /home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/externals/joblib/__pycache__/parallel.cpython-35.pyc matches
Binary file /home/r2d2/anaconda2/pkgs/scikit-learn-0.20.0-py35h4989274_1/lib/python3.5/site-packages/sklearn/externals/joblib/__pycache__/_dask.cpython-35.pyc matches

Upvotes: 1

Related Questions