CODEWITHSUNDEEP
pythontensorflow
user5739619
user5739619

Reputation: 1838

Tensorflow 2.3.0 in Ubuntu: libcublas.so.10: cannot open shared object file

I was trying to run in Keras but got

terminate called after throwing an instance of 'std::bad_alloc'

which doesn't make sense since I'm running the same Unet as before. I did make changes to CUDA, so I'm guessing that's the cause of this

Whenever I use tensorflow (I use version 2.3.0 in Ubuntu 16 with an NVIDIA GPU) and try

gpus = tf.config.experimental.list_physical_devices('GPU')

it shows gpus as an empty list and says

Successfully opened dynamic library libcudart.so.10.1
2020-09-14 16:39:11.975096: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcublas.so.10'; dlerror: libcublas.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.0/lib64:/usr/local/cuda-10.0/lib64::/usr/local/cuda-10.0/lib64::/usr/local/cuda-11.0/lib64::/usr/local/cuda-11.0/lib64
2020-09-14 16:39:11.975158: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-09-14 16:39:11.975197: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-09-14 16:39:11.975232: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-09-14 16:39:11.975380: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcusparse.so.10'; dlerror: libcusparse.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.0/lib64:/usr/local/cuda-10.0/lib64::/usr/local/cuda-10.0/lib64::/usr/local/cuda-11.0/lib64::/usr/local/cuda-11.0/lib64
2020-09-14 16:39:11.975436: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7

even though I set

export PATH=/usr/local/cuda-10.0/bin${PATH:+:${PATH}}
export PATH=/usr/local/cuda-10.0/bin:/usr/local/cuda-10.0/NsightCompute-1.0${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64:${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

and which nvcc shows

/usr/local/cuda-10.0/bin/nvcc

and $LD_LIBRARY_PATH

shows

bash: /usr/local/cuda-10.0/lib64::/usr/local/cuda-11.0/lib64:/usr/local/cuda-11.0/extras/CUPTI/lib64:/usr/local/cuda/lib64:/usr/local/cuda-11.0/extras/CUPTI/lib64:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda-11.0/lib64: No such file or directory

and ~/.bashrc shows

export PATH="$PATH:/usr/local/cuda-10.0/bin"
export LD_LIBRARY_PATH="/usr/local/cuda-10.0/lib64"${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

can anyone help?

EDIT

Output of sudo find / -name "libcublas*" is below:

/usr/share/doc/libcublas7.5
/usr/share/doc/libcublas-11-0
/usr/share/lintian/overrides/libcublas7.5
/usr/share/man/man7/libcublas.so.7.gz
/usr/share/man/man7/libcublas.7.gz
/usr/local/MATLAB/R2018a/bin/glnxa64/libcublas.so.9.0.176
/usr/local/MATLAB/R2018a/bin/glnxa64/libcublas.so.9.0
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcublas.so.11
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcublasLt.so.11
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcublas.so.11.2.0.252
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcublasLt.so.11.2.0.252
/usr/local/cuda-10.0/doc/man/man7/libcublas.so.7
/usr/local/cuda-10.0/doc/man/man7/libcublas.7
/usr/local/cuda-10.0/targets/x86_64-linux/lib/stubs/libcublas.so
/usr/local/cuda-10.0/targets/x86_64-linux/lib/libcublas.so
/usr/local/cuda-10.0/targets/x86_64-linux/lib/libcublas.so.10.0.130
/usr/local/cuda-10.0/targets/x86_64-linux/lib/libcublas_static.a
/usr/local/cuda-10.0/targets/x86_64-linux/lib/libcublas.so.10.0
/usr/lib/x86_64-linux-gnu/libcublas.so.7.5.18
/usr/lib/x86_64-linux-gnu/stubs/libcublas.so
/usr/lib/x86_64-linux-gnu/libcublas.so
/usr/lib/x86_64-linux-gnu/libcublas.so.7.5
/usr/lib/x86_64-linux-gnu/libcublas_device.a
/usr/lib/x86_64-linux-gnu/libcublas_static.a
find: ‘/run/user/1000/gvfs’: Permission denied
/home/me/.julia/packages/CuArrays/clDeS/src/blas/libcublas_types.jl
/home/me/.julia/packages/CuArrays/clDeS/src/blas/libcublas.jl
/home/me/Downloads/pgilinux-2019-1910-x86-64/install_components/linux86-64-nollvm/19.10/lib/libcublas.ipl
/home/me/Downloads/pgilinux-2019-1910-x86-64/install_components/linux86-64-nollvm/19.10/lib/libcublasemu.so
/home/me/Downloads/pgilinux-2019-1910-x86-64/install_components/linux86-64-nollvm/19.10/lib/libcublasemu.a
/home/me/Downloads/pgilinux-2019-1910-x86-64/install_components/linux86-64-nollvm/19.10/REDIST/libcublasemu.so
/home/me/Downloads/pgilinux-2019-1910-x86-64/install_components/linux86-64-llvm/19.10/lib/libcublas.ipl
/home/me/Downloads/install_components/linux86-64-nollvm/19.10/lib/libcublas.ipl
/home/me/Downloads/install_components/linux86-64-nollvm/19.10/lib/libcublasemu.so
/home/me/Downloads/install_components/linux86-64-nollvm/19.10/lib/libcublasemu.a
/home/me/Downloads/install_components/linux86-64-nollvm/19.10/REDIST/libcublasemu.so
/home/me/Downloads/install_components/linux86-64-llvm/19.10/lib/libcublas.ipl
/opt/pgi/linux86-64-nollvm/19.10/lib/libcublas.ipl
/opt/pgi/linux86-64-nollvm/19.10/lib/libcublasemu.so
/opt/pgi/linux86-64-nollvm/19.10/lib/libcublasemu.a
/opt/pgi/linux86-64-nollvm/19.10/REDIST/libcublasemu.so
/opt/pgi/linux86-64-nollvm/2019/cuda/9.2/lib64/libcublas.so.9.2.113
/opt/pgi/linux86-64-nollvm/2019/cuda/9.2/lib64/libcublas.so
/opt/pgi/linux86-64-nollvm/2019/cuda/9.2/lib64/libcublas_device.a
/opt/pgi/linux86-64-nollvm/2019/cuda/9.2/lib64/libcublas_static.a
/opt/pgi/linux86-64-nollvm/2019/cuda/9.2/lib64/libcublas.so.9.2
/opt/pgi/linux86-64-nollvm/2019/cuda/10.1/lib64/libcublasLt.so.10
/opt/pgi/linux86-64-nollvm/2019/cuda/10.1/lib64/libcublasLt_static.a
/opt/pgi/linux86-64-nollvm/2019/cuda/10.1/lib64/libcublas.so
/opt/pgi/linux86-64-nollvm/2019/cuda/10.1/lib64/libcublasLt.so.10.2.1.243
/opt/pgi/linux86-64-nollvm/2019/cuda/10.1/lib64/libcublas.so.10.2.1.243
/opt/pgi/linux86-64-nollvm/2019/cuda/10.1/lib64/libcublasLt.so
/opt/pgi/linux86-64-nollvm/2019/cuda/10.1/lib64/libcublas_static.a
/opt/pgi/linux86-64-nollvm/2019/cuda/10.1/lib64/libcublas.so.10
/opt/pgi/linux86-64-nollvm/2019/cuda/10.0/lib64/libcublas.so
/opt/pgi/linux86-64-nollvm/2019/cuda/10.0/lib64/libcublas.so.10.0.130
/opt/pgi/linux86-64-nollvm/2019/cuda/10.0/lib64/libcublas_static.a
/opt/pgi/linux86-64-nollvm/2019/cuda/10.0/lib64/libcublas.so.10.0
/opt/pgi/linux86-64-llvm/19.10/lib/libcublas.ipl
/opt/pgi/linux86-64-llvm/2019/cuda/9.2/lib64/libcublas.so.9.2.113
/opt/pgi/linux86-64-llvm/2019/cuda/9.2/lib64/libcublas.so
/opt/pgi/linux86-64-llvm/2019/cuda/9.2/lib64/libcublas_device.a
/opt/pgi/linux86-64-llvm/2019/cuda/9.2/lib64/libcublas_static.a
/opt/pgi/linux86-64-llvm/2019/cuda/9.2/lib64/libcublas.so.9.2
/opt/pgi/linux86-64-llvm/2019/cuda/10.1/lib64/libcublasLt.so.10
/opt/pgi/linux86-64-llvm/2019/cuda/10.1/lib64/libcublasLt_static.a
/opt/pgi/linux86-64-llvm/2019/cuda/10.1/lib64/libcublas.so
/opt/pgi/linux86-64-llvm/2019/cuda/10.1/lib64/libcublasLt.so.10.2.1.243
/opt/pgi/linux86-64-llvm/2019/cuda/10.1/lib64/libcublas.so.10.2.1.243
/opt/pgi/linux86-64-llvm/2019/cuda/10.1/lib64/libcublasLt.so
/opt/pgi/linux86-64-llvm/2019/cuda/10.1/lib64/libcublas_static.a
/opt/pgi/linux86-64-llvm/2019/cuda/10.1/lib64/libcublas.so.10
/opt/pgi/linux86-64-llvm/2019/cuda/10.0/lib64/libcublas.so
/opt/pgi/linux86-64-llvm/2019/cuda/10.0/lib64/libcublas.so.10.0.130
/opt/pgi/linux86-64-llvm/2019/cuda/10.0/lib64/libcublas_static.a
/opt/pgi/linux86-64-llvm/2019/cuda/10.0/lib64/libcublas.so.10.0
/var/lib/dpkg/info/libcublas-11-0.md5sums
/var/lib/dpkg/info/libcublas-11-0.list
/var/lib/dpkg/info/libcublas7.5:amd64.list
/var/lib/dpkg/info/libcublas7.5:amd64.triggers
/var/lib/dpkg/info/libcublas7.5:amd64.md5sums
/var/lib/dpkg/info/libcublas7.5:amd64.shlibs
/var/lib/dpkg/info/libcublas7.5:amd64.symbols

Upvotes: 3

Views: 7118

Answers (1)

SkyDriver
SkyDriver

Reputation: 65

I had the same problem, I went to the https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ and downloaded and installed libcublas10_10.1.0.105-1_amd64.deb

Upvotes: 3

Related Questions