No gpu support with tf-nightly 2.4.0 and I get "Could not load dynamic library errors"

Question

I am doing the TF tutorial at https://www.tensorflow.org/tutorials/images/classification and I am not able to get gpu support and I get "could not load dynamic library errors". I will paste errors below.

my OS is Ubuntu 18.04 I followed the guide on the tensorflow website for installing gpu, but still had problems, so I tried again using conda to help with installing the cuda files.

Before starting the tutorial I created a new conda environment and conda installed:

cudatoolkit 10.1.243 h6bb024c_0
cudnn 7.6.5 cuda10.1_0
cupti 10.1.168 0
python 3.8.5 h7579374_1

Conda does not install tf-nightly, so then I followed the tutorial and pip installed: tf-nightly 2.4.0.dev20200925 pypi_0 pypi

PROBLEM1, here is what I get when I test to see if a gpu is available:

>>> import tensorflow as tf
>>> tf.test.is_gpu_available(
...     cuda_only=False, min_cuda_compute_capability=None
... )


2020-09-25 14:16:34.733174: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2020-09-25 14:16:34.733394: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-25 14:16:34.734906: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6705GHz coreCount: 28 deviceMemorySize: 10.91GiB deviceMemoryBandwidth: 451.17GiB/s
2020-09-25 14:16:34.735139: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2020-09-25 14:16:34.735278: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcublas.so.11'; dlerror: libcublas.so.11: cannot open shared object file: No such file or directory
2020-09-25 14:16:34.735313: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-09-25 14:16:34.735340: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-09-25 14:16:34.735367: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-09-25 14:16:34.735484: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcusparse.so.11'; dlerror: libcusparse.so.11: cannot open shared object file: No such file or directory
2020-09-25 14:16:34.735517: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-09-25 14:16:34.735535: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1753] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2020-09-25 14:16:34.735568: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-09-25 14:16:34.735584: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0 
2020-09-25 14:16:34.735599: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N 
False

PROBLEM 2, I checked the folders inside my conda env for the missing libraries mentioned above. Here is what I get after I search my machine for the "missing" libraries from the error messages above and I copy those libraries into ".../anaconda3/envs/tfnight2/lib/python3.8/site-packages/tensorflow/include/tensorflow/stream_executor/platform/default":

>>> import tensorflow as tf

2020-09-25 12:06:53.393728: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2020-09-25 12:06:53.393789: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.

Answer 1

For the newest version of tf-nightly you should be using cuda 11.0, if you install it alongside cuDNN 8.0, your problems should disappear.