CODEWITHSUNDEEP
dockertensorflowcudanvidia
Danail
Danail

Reputation: 10583

Tensorflow/nvidia/cuda docker mismatched versions

I am trying to use tensorflow and nvidia with docker, but hitting the following error:

docker run --runtime=nvidia -it --rm tensorflow/tensorflow:latest-gpu python -c "import tensorflow as tf; tf.enable_eager_execution(); print(tf.reduce_sum(tf.random_normal([1000, 1000])))"

docker: Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container process caused "process_linux.go:424: container init caused \"process_linux.go:407: running prestart hook 1 caused \\"error running hook: exit status 1, stdout: , stderr: exec command: [/usr/bin/nvidia-container-cli --load-kmods configure --ldconfig=@/sbin/ldconfig.real --device=all --compute --utility --require=cuda>=10.0 brand=tesla,driver>=384,driver<385 --pid=5393 /var/lib/docker/overlay2/......./merged]\\nnvidia-container-cli: requirement error: unsatisfied condition: brand = tesla\\n\\"\"": unknown.

I get similar error when trying to run nvidia-smi:

docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi

but when trying to run nvidia-smi with cuda:9.0-base, it works like a charm:

docker run --runtime=nvidia --rm nvidia/cuda:9.0-base nvidia-smi

Do I need to make sure that cuda 10 works or I can run tensorflow with cuda 9? And how can I run the docker image of tensorflow with cuda:9.0-base? (still a docker newby).

Thanks a lot!

Upvotes: 3

Views: 2504

Answers (1)

Danail
Danail

Reputation: 10583

Ok I think I am finally starting to figure out the mess on my machine.

The tensorflow image does NOT care about the cuda image version, it does not use the docker cuda image. It cares about my nvidia drivers, since it has CUDA integrated in the tensorflow image.

(The docker cuda image that is working with my current drivers, is cuda:9.0)

That meant I have to find a tensorflow image that is working with my drivers (390.116), or update the drivers.

I tried the same command with tensorflow:1.12.0-gpu-py3, and it didn't have any problems.

Upvotes: 5

Related Questions