CODEWITHSUNDEEP
cudatensorflowgpucudnn
Michael
Michael

Reputation: 149

multi process multi GPU with tensorflow, windows

I'm little bit new with tensor-flow.. so please be gentle with me.. I have problem with creating second process that load tensorflow on already working GPU.

the error I get is: 

\cuda\cuda_dnn.cc:385] could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
\cuda\cuda_dnn.cc:392] error retrieving driver version: Permission denied: could not open driver version path for reading: /proc/driver/nvidia/version
\cuda\cuda_dnn.cc:352] could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM
\kernels\conv_ops.cc:532] Check failed: stream->parent()->GetConvolveAlgorithms(&algorithms)
\cuda\cuda_dnn.cc:385] could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED

Hardware details :

super micro -  4028GR-TRT
8 GPU's 1080 
CUDA: 8 
cudnn: 5.1 
windows: 10 
tensorflow: 0.12.1 / 1.0.1

My PC shouldn't be a problem 

windows 7
gpu 1070
cuda 8
cudnn 5.1 
tensorflow 0.12.1

Can someone tell me why on my PC everything is ok but not on the big one(supermicro)?

is this windows / driver issues maybe?

I try to update NVIDIA driver.. no help on that ..

Upvotes: 2

Views: 1493

Answers (1)

mrry
mrry

Reputation: 126154

TensorFlow is not always good at sharing GPUs with other processes (including other instances of itself!). The typical workaround is to use the %CUDA_VISIBLE_DEVICES% environment variable to prevent the two processes from clashing over the same GPU. For example:

C:\>set CUDA_VISIBLE_DEVICES=0
C:\>python tensorflow_program_1.py

While in another command prompt you could tell TensorFlow to use a different GPU as follows:

C:\>set CUDA_VISIBLE_DEVICES=1
C:\>python tensorflow_program_2.py

Upvotes: 2

Related Questions