GridSearchCV with n_jobs = -1 kernel Restarts

Question

I'm running a Jupyter Notebook in a Azure machine with the following settings :

• OS: Ubuntu 18.04.5 LTS
• Memory: 64Gb
• Number of cores: 16
• Anaconda Environment
• Architecture: x86_64
• CPU op-mode(s): 32-bit, 64-bit
• Byte Order: Little Endian
• CPU(s): 16
• On-line CPU(s) list: 0-15
• Thread(s) per core: 2
• Core(s) per socket: 8
• Socket(s): 1
• NUMA node(s): 2
• Vendor ID: AuthenticAMD
• CPU family: 23
• Model: 49
• Model name: AMD EPYC 7452 32-Core Processor
• Stepping: 0
• CPU MHz: 2345.605
• BogoMIPS: 4691.21
• Hypervisor vendor: Microsoft
• Virtualization type: full
• L1d cache: 32K
• L1i cache: 32K
• L2 cache: 512K
• L3 cache: 16384K
• NUMA node0 CPU(s): 0-7
• NUMA node1 CPU(s): 8-15

I'm trying to launch a GridSearchCV like this:

# DecissionTreeClassifier
arbol = DTC()
grid_arbol = {"max_depth":list(range(5,15)),
                "class_weight":["balanced"]}
gs_arbol = GridSearchCV(arbol,
                        grid_arbol,
                        cv=2,
                        scoring=scoring,
                        verbose=4,
                        n_jobs=-1) # -1 should use all cores (16)
grid_search.fit(X_training, y_training)

Once I launch this code, Kernel restarts. I have to change the n_jobs=8 (half of the total cores) to launch correctly this GridSearchCV.

The question is that I would like to use all CPU cores (and all the run memory), but I don't find the clue to use them from Ubuntu.

Is there any OS/Jupyter/Python/Anaconda option to launch with all n_jobs?

Thanks in advance

Answer 1

I ran into this issue when searching for a solution for what seems to be the same problem I'm having when fitting a LassoCV. Whenever I specify n_jobs=-1 I get a kernel crash. I'm using VS Code with Jupyter extension on a 2018 6-core i7 MacBook Pro. Everything is cool when I use n_jobs=5 or less.

I found this resource that seems to explain the problem.

Why do I sometime get a crash/freeze with n_jobs > 1 under OSX or Linux?