CODEWITHSUNDEEP
theanotheano-cuda
sachinruk
sachinruk

Reputation: 9869

theanorc for multiple GPUs

I have a aws machine with 4 GPUs:

00:03.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K520] (rev a1)
00:04.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K520] (rev a1)
00:05.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K520] (rev a1)
00:06.0 VGA compatible controller: NVIDIA Corporation GK104GL [GRID K520] (rev a1)

and my theanorc file looks like this:

[global]
floatX = float32
device = gpu0

[lib]
cnmem = 1

When I open one jupyter notebook and import theano I get the following (which I assume is only using one GPU):

Using Theano backend.
Using gpu device 0: GRID K520 (CNMeM is enabled with initial size: 95.0% of memory, cuDNN 5105)
/home/sabeywardana/anaconda3/lib/python3.5/site-packages/theano/sandbox/cuda/__init__.py:600: UserWarning: Your cuDNN version is more recent than the one Theano officially supports. If you see any problems, try updating Theano or downgrading cuDNN to version 5.

However, if I open a second jupyter notebook on the same machine at the same time. Then I get the error:

ERROR (theano.sandbox.cuda): ERROR: Not using GPU. Initialisation of device 0 failed:
initCnmem: cnmemInit call failed! Reason=CNMEM_STATUS_OUT_OF_MEMORY. numdev=1

ERROR (theano.sandbox.cuda): ERROR: Not using GPU. Initialisation of device gpu failed:
initCnmem: cnmemInit call failed! Reason=CNMEM_STATUS_OUT_OF_MEMORY. numdev=1

If I manually change my .theanorc to use gpu1 then the second jupyter notebook works fine. So the question is: Is there a way to configure .theanorc to just get the available GPU?

Upvotes: 0

Views: 955

Answers (2)

Pascal Lamblin
Pascal Lamblin

Reputation: 356

You can use device=gpu, which will select the first available GPU. However, in your case, GPU 0 will still be considered "available" (it does not have much memory left, but execution is still possible). You can use nvidia-smi to set the compute mode of your GPUs to "Exclusive Thread", so that the first notebook "blocks" the first GPU for its exclusive use, and the second notebook will use another one.

Another option is to change the THEANO_FLAGS environment variable from inside the notebook, before importing theano. Something like:

import os

os.environ['THEANO_FLAGS'] = os.environ.get('THEANO_FLAGS', '') + ',' + 'device=gpu1'

import theano

Upvotes: 2

Avijit Dasgupta
Avijit Dasgupta

Reputation: 2065

It is not possible to change the gpu device after importing theano.

May be you can try this-

import os
os.system("THEANO_FLAGS='device=gpu0' python script_1.py")
os.system("THEANO_FLAGS='device=gpu1' python script_2.py")
os.system("THEANO_FLAGS='device=gpu1' python script_3.py")
os.system("THEANO_FLAGS='device=gpu1' python script_4.py")

If you want to do it from inside the notebook (more programmatical) you may use following snippet :-

import theano.sandbox.cuda
theano.sandbox.cuda.use("gpu0")

Paste this to every notebook and change the gpu id. It will work.

Upvotes: 1

Related Questions