ImportError: libcublas.so.9.0: cannot open shared object file

Question

currently I have cuda 8.0 and cuda 9.0 installed in Gpu support system. I ran into this error while importing from keras module. It says like failed to load native tensorflow runtime. The error log which i received was:

Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
from tensorflow.python.pywrap_tensorflow_internal import *
File "/usr/local/lib/python3.5/dist-
packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
_pywrap_tensorflow_internal = swig_import_helper()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "/usr/lib/python3.5/imp.py", line 242, in load_module
return load_dynamic(name, filename, file)
File "/usr/lib/python3.5/imp.py", line 342, in load_dynamic
return _load(spec)

ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "Try1.py", line 11, in <module>
from keras.models import Sequential
File "/usr/local/lib/python3.5/dist-packages/Keras-2.0.9-py3.5.egg/keras/__init__.py", line 3, in <module>
File "/usr/local/lib/python3.5/dist-packages/Keras-2.0.9-py3.5.egg/keras/utils/__init__.py", line 6, in <module>
File "/usr/local/lib/python3.5/dist-packages/Keras-2.0.9-py3.5.egg/keras/utils/conv_utils.py", line 3, in <module>
File "/usr/local/lib/python3.5/dist-packages/Keras-2.0.9-py3.5.egg/keras/backend/__init__.py", line 83, in <module>
File "/usr/local/lib/python3.5/dist-packages/Keras-2.0.9-py3.5.egg/keras/backend/tensorflow_backend.py", line 1, in <module>
File "/usr/local/lib/python3.5/dist-packages/tensorflow/__init__.py", line 24, in <module>
from tensorflow.python import *
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/__init__.py", line 49, in <module>
from tensorflow.python import pywrap_tensorflow
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 73, in <module>
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
from tensorflow.python.pywrap_tensorflow_internal import *
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
_pywrap_tensorflow_internal = swig_import_helper()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "/usr/lib/python3.5/imp.py", line 242, in load_module
return load_dynamic(name, filename, file)
File "/usr/lib/python3.5/imp.py", line 342, in load_dynamic
return _load(spec)
ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory


Failed to load the native TensorFlow runtime.

When I run nvcc --version, the cuda version returned is,

Cuda compilation tools, release 8.0, V8.0.61

I read about some similar post but couldn't solve my issue. Mostly I think this is a clash between two cuda versions. Can anyone tell me how to solve this?

Answer 1

This happens because the system is looking for the appropriate cuda version to run the training. I solved this in a simple way in conda.

Go to this link and find the cuda version for your tensorflow.

Once you find, execute this.

Search for the specific cudatoolkit version by

conda search cudatoolkit
conda install cudatoolkit=*give_your_version_number_here*

Next install the corresponding cudnn

search by conda search cudnn, this will show the corresponding cudnn for your cudatoolkit you just installed.

finally just install,

conda install cudnn=*your_version_number*

P.S -> That's the beauty of conda. Don't get your hands dirty, when you don't havta ;)

Answer 2

I just had this same problem on Google Colab, it seems we have to install Cuda 9.0 do overcome it. (Cuda 9.2 did not worked)

Just follow:

!wget https://developer.nvidia.com/compute/cuda/9.0/Prod/local_installers/cuda-repo-ubuntu1704-9-0-local_9.0.176-1_amd64-deb

!ls  # Check if required cuda 9.0 amd64-deb file is downloaded

!dpkg -i cuda-repo-ubuntu1704-9-0-local_9.0.176-1_amd64-deb

!ls /var/cuda-repo-9-0-local | grep .pub

!apt-key add /var/cuda-repo-9-0-local/7fa2af80.pub

!apt-get update

!sudo apt-get install cuda-9.0

credits to Medium Post: Sifat Muhammad Abdullah

Answer 3

I solved by following steps:

# Add NVIDIA package repository
sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_9.1.85-1_amd64.deb
sudo apt install ./cuda-repo-ubuntu1604_9.1.85-1_amd64.deb
wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/nvidia-machine-learning-repo-ubuntu1604_1.0.0-1_amd64.deb
sudo apt install ./nvidia-machine-learning-repo-ubuntu1604_1.0.0-1_amd64.deb
sudo apt update

# Install CUDA and tools. Include optional NCCL 2.x
sudo apt install cuda9.0 cuda-cublas-9-0 cuda-cufft-9-0 cuda-curand-9-0 \
    cuda-cusolver-9-0 cuda-cusparse-9-0 libcudnn7=7.2.1.38-1+cuda9.0 \
    libnccl2=2.2.13-1+cuda9.0 cuda-command-line-tools-9-0

# Optional: Install the TensorRT runtime (must be after CUDA install)
sudo apt update
sudo apt install libnvinfer4=4.1.2-1+cuda9.0

Answer 4

I was trying to install tf in an Anaconda environment and had the same issue. After install CUDA and cuDNN I had to make a new environment first and then in that env pip install tensorflow-gpu

Answer 5

You will need to update your LD_LIBRARY_PATH, so that it points to the /usr/local/cuda-9.0/lib64. Add the following line to your .bashrc file (or any other terminal you use)

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-9.0/lib64/

Answer 6

I had this problem after upgrading from Ubuntu 16.04 to 18.04. I had installed CUDA 9.1 with Tensorflow 1.12.0, and that version of Tensorflow needs CUDA 9.0. When I tried installing CUDA 9.0, it wrecked my video drivers.

Luckily, I found instructions for installing multiple CUDA libraries. The only parts I needed so far are to download the installer script, run it with the --silent --toolkit --override options, and then set LD_LIBRARY_PATH=/usr/local/cuda-9.0/lib64. I also set the symbolic link at /usr/local/cuda back to /usr/local/cuda-9.1, but I'm not sure if that's needed.

So far, Tensorflow is working fine.

Answer 7

Whenever you install a new version of cuda, you need to check the permanent path that may have been added.

to check this in the terminal run

gedit ~/.bashrc

if you see that the old cuda path is there, you need to change it to the new one.

for example mine was:

export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64:/usr/lib/nvidia-384

I changed both the cuda path and the nvidia path since I also upgraded the driver to:

export LD_LIBRARY_PATH=/usr/local/cuda-9.0/lib64:/usr/lib/nvidia-390

Hope that helps :)

Answer 8

Tensorflow version >= 1.5 requires CUDA version > 8.0. So if you have CUDA version 8.0, you can downgrade your tensorflow version to 1.4.

pip install tensorflow-gpu==1.4

Answer 9

I've got that error but it was related to lack of GPU memory although the error message seemed not directly related. I've tried to reboot the server, as other suggested, but I've only gained few more iterations (epochs). To solve the problem, I've upgraded the GPU server instance for a larger one from my cloud provider still using the same server image.

I hope that can help someone else.

Answer 10

This issue is related to Google's protobuf-compiler due to which tensorflow fails to find the shared object file, in this instance, libcublas.so.9.0.

Building the latest version of protobuf (3.5.0) from source didn't help either. What worked for me was to install the system-wide protobuf compiler through apt install protobuf-compiler on Ubuntu 16.04. And, install the python version through pip3 install protobuf. I am using CUDA 9.0 as 9.1 is not yet compatible with tensorflow's pre-built binary.

You can check the system-wide protobuf version using protoc --version which is 2.6.1 on 16.04. The protoc python version is 3.5.2.post1. Hope this helps. I had a similar issue using earlier versions of tensorflow and CUDA 8, and had documented this troubleshooting procedure. Using the same procedure, I am able to use tensorflow 1.8.0 too.

Answer 11

The Tensorflow Binaries do not support Cuda 9.1 as of now. So the solution might be just installing Tensorflow from source.

I too had the above mentioned issue while I had installed Cuda 9.1+cuDNN 6 for Tensorflow and this is what worked for me.

https://devtalk.nvidia.com/default/topic/1026198/cuda-setup-and-installation/cuda-9-0-importerror-libcublas-so-8-0

Additionally, you have the following two options:

1. CUDA 8.0 + Tensorflow - the most stable release of CUDA that works with Tensorflow Binaries.

2. CUDA 9.0 + Tensorflow - latest CUDA version that Tensorflow Binaries are compatible with.

Answer 12

Try the following code

 pip3 install --upgrade tensorflow-gpu==1.4

After you typing this command pip3 install --upgrade tensorflow-gpu==1.4 in the terminal, the tensorflow will downgrade to 1.4.0. This bug caused by tensorflow 1.6.0.