CODEWITHSUNDEEP
pythonmultithreadingjupyter-labcpu-cores
Reut
Reut

Reputation: 1592

Control number of CPU using in jupyterlab server

I'm using jupyterlab and I know that I have 12 cores available. At the moment I use only 1 and I would like to use more. I have tried to changed the number I use by write this in the terminal:

export JULIA_NUM_THREADS=7

but then when I print:

import threading
threading.activeCount()
>>>5

how can I make more CPU available for my jupyterlab notebook? This is really not my field so I'm sorry if is smething really simple I just don't understand what am I doing wrong and where to start from.

Upvotes: 2

Views: 4142

Answers (2)

Iñigo González
Iñigo González

Reputation: 3945

TLDD; No configuration needed. It is available to you, just need to code explicitely what you want to run in parallel.

JULIA_ACTIVE_THREADS is a configuration option for the Julia Kernel in Jupyter, not for the Python Kernel (the process that runs your notebook code).

Unless you run Jupyter inside a container, you can use out of the box all cores available in your system. If Jupyter is in a container or a virtual machine, it will use what you allocate and nothing more.

Just remember that by default you use 1 core when you run your Jupyter kernel.

When you run threading.active_count() and get 1, this means you are using one running thread on your code. Moden processors can use several threads for each available core. The bad news is that this is not a measure about how good you are using the cpu.

Python can act as an orchestrator for libraries that work in paraller behind the scenes (think numpy, pandas, tensorflow...).

If you want to code Python code that use more than 1 thread and/or 1 CPU, take a look at the multiprocess module.

The multipreocessing module is part of the standard library, and you can use it inside without trouble inside Jupyter. Probably you will find the Process and Pool methods useful (if you want to work with deep learning, there is a pytorch.multiprocessing module with the same interface but with support for working with GPUs in different threads).

Upvotes: 1

D-E-N
D-E-N

Reputation: 1272

A few thoughts, but to long for a comment, i am not familiar with jupyter, only "normal python", so maybe this all gets in the wrong direction ;):

  1. As far as i know, the àctive_count (in my opinion you should not use the old camelCase name) only returns the amount of active threads, not the available. So try to add more threads. I have a Quadcore and jupyter starts with 5 threads, but i can add more.
  2. Multithreading is not the same as multiprocessing (If you want to run on different Cores you have to use multiprocessing) (python thread vs. multiproccess), maybe you are looking for the wrong thing?

Upvotes: 0

Related Questions