CUDA Tensorflow Version ,nvidia-smi version issue. Failed to get convolution algorithm. This is probably because cuDNN failed to initialize,

Question

Tried, preds = model.predict(k[np.newaxis,...])

UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[node model/stem_conv/Conv2D (defined at :3) ]] [Op:__inference_distributed_function_18348]

Hardware: Make: OMEN, OS_ Windows 10, GPU NVIDIA GEFORCE RTX 2060, My System configuration

!nvcc --version

> nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2019 NVIDIA
> Corporation Built on Sun_Jul_28_19:12:52_Pacific_Daylight_Time_2019
> Cuda compilation tools, release 10.1, V10.1.243

!nvidia-smi

Mon Jul 20 23:15:20 2020

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 451.77       Driver Version: 451.77       CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 2060   WDDM  | 00000000:01:00.0 Off |                  N/A |
| N/A   38C    P8     5W /  N/A |   5304MiB /  6144MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A     18900      C   ...nvs\tensorflow\python.exe    N/A      |
+-----------------------------------------------------------------------------+

Seems like some thing is missing. CUDA version is not in Sync is what I guess. Please correct me. If its NVIDIA STUDIO DRIVER not appropriate, Could some one please help me the one appropriate for Tensorflow 2.1.0

Tensor Flow Version: 2.1.0, Keras Version: 2.2.4-tf , Python 3.7.7 (default, May 6 2020, 11:45:54) [MSC v.1916 64 bit (AMD64)] ,Pandas 1.0.5 ,Scikit-Learn 0.23.1

Answer 1

I had similar issue. It got fixed after I downgraded CUDA from 11 to 10.1 version (NVIDIA Link).

As per TF documentation:

CUDA® Toolkit —TensorFlow supports CUDA 10.1 (TensorFlow >= 2.1.0)

EDIT (more info below):

You can find the compatible driver at: https://www.nvidia.com/en-us/drivers/results/149127/

I use Python 3.7 along with the tensorflow-gpu installed in a conda environment.

NOTE: If you installed TF with pip install tensorflow, it may not have the required python packages for cuda downloaded. I would recommend you uninstall/re-install with:

pip install tensorflow-gpu

Here's my output for a quick test:

import tensorflow as tf
print(tf.__version__)
print(tf.config.experimental.list_physical_devices('GPU'))

2.1.0
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

For your reference:

nvidia-smi

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 431.70       Driver Version: 431.70       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 206... WDDM  | 00000000:01:00.0  On |                  N/A |
| 32%   44C    P8    21W / 175W |    510MiB /  8192MiB |      3%      Default |
+-------------------------------+----------------------+----------------------+

nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Fri_Feb__8_19:08:26_Pacific_Standard_Time_2019
Cuda compilation tools, release 10.1, V10.1.105

From cudnn.h (Path: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\include)

Please note that you need to manually move the files to this location. Please refer to the installation instructions:

Download: https://developer.nvidia.com/compute/machine-learning/cudnn/secure/7.6.5.32/Production/10.1_20191031/cudnn-10.1-windows10-x64-v7.6.5.32.zip

Instructions: https://docs.nvidia.com/deeplearning/sdk/cudnn-install/index.html#install-windows

#define CUDNN_MAJOR 7
#define CUDNN_MINOR 6
#define CUDNN_PATCHLEVEL 5

Hope this helps!