CODEWITHSUNDEEP
pythonpytorchgpu
ESH64
ESH64

Reputation: 421

RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED using pytorch

I am trying to run a simple pytorch sample code. It's works fine using CPU. But when using GPU, i get this error message:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 263, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 260, in _conv_forward
    self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

The code i am trying to run is the following:

import torch
from torch import nn
m = nn.Conv1d(16, 33, 3, stride=2)
m=m.to('cuda')
input = torch.randn(20, 16, 50)
input=input.to('cuda')
output = m(input)

I am running this code in a NVIDIA docker with CUDA version 10.2 and my GPU is a RTX 2070

Upvotes: 32

Views: 97376

Answers (10)

Marina Thalassini
Marina Thalassini

Reputation: 21

I had the same issue. Turns out multiple processes were trying to run at the same time because by using control + C not all the processes are being terminated. I logout and in on the server and it was working.

Upvotes: 1

Ruibin
Ruibin

Reputation: 11

My solution is similar to saturn660's answer and the link provided there is also helpful to understand the problem.

For many users, they might install pytorch using conda or pip directly without specifying any labels, e.g. pip install torch. It might work for some users but can fail if the cuda version doesn't match the official default build.

If you check the pytorch install guide, it actually instructs the users to provide --index-url https://download.pytorch.org/whl/cuxxx where xxx stands for a cuda version like 118 for cuda 11.8, for installing from pip. But that's the way to install the latest stable version matching the specified cuda version. If you need to install a pytorch version that matches a cuda version you can use, you can (for example):

pip install torch==1.8.1+cu102 -f https://download.pytorch.org/whl/torch_stable.html

It means you want to install the torch 1.8.1 version built for cuda 10.2 which you can access in your computer. Of course, you need to download the corresponding cuDNN library and extract it to the cuda 10.2's lib64 and include directories.

Check this page for all the combinations you can install.

Upvotes: 0

Tony.sy
Tony.sy

Reputation: 17

Sometimes, if any error happens in the CUDA c++ code that is converted into .so file and used inside Python code, it could cause this problem, so check your C++ source code if you have any.

Upvotes: 0

sathish kumar
sathish kumar

Reputation: 26

In my problem i used to kill exisiting process in gpu.Use nvidia-smi to check what are the process are running.Use killall -9 python3(what process you want) to kill process.After freeup space then run the process.

Upvotes: 1

Naeem Iqbal
Naeem Iqbal

Reputation: 11

In my case, I had an array indexing operation but the index was out of bounds. CUDA did not tell me that. I was using inference on a neural network. So I moved to CPU instead of the GPU. The logs were much more informative after that. For debugging if you see this error, switch to CPU first and you will know what to do.

Upvotes: 1

Flying Fox
Flying Fox

Reputation: 21

I had the same issue when I was training yolov7 with a chess dataset. By reducing batch size from 8 to 4, the issue was solved.

Upvotes: 2

Ioannis Nasios
Ioannis Nasios

Reputation: 8527

In my cases this error occurred when trying to estimate loss. I used a mixed bce-dice loss. It turned out that my output was linear instead of sigmoid. I then used the sigmoid predictions as of bellow and worked fine.

output = torch.nn.Sigmoid()(output)
loss = criterion1(output, target)

Upvotes: 1

creiser
creiser

Reputation: 401

In my case it actually had nothing do with the PyTorch/CUDA/cuDNN version. PyTorch initializes cuDNN lazily whenever a convolution is executed for the first time. However, in my case there was not enough GPU memory left to initialize cuDNN because PyTorch itself already held the entire memory in its internal cache. One can release the cache manually with "torch.cuda.empty_cache()" right before the first convolution that is executed. A cleaner solution is to force cuDNN initialization at the beginning by doing a mock convolution:

def force_cudnn_initialization():
    s = 32
    dev = torch.device('cuda')
    torch.nn.functional.conv2d(torch.zeros(s, s, s, s, device=dev), torch.zeros(s, s, s, s, device=dev))

Calling the above function at the very beginning of the program solved the problem for me.

Upvotes: 28

saturn660
saturn660

Reputation: 323

There is some discussion regarding this here. I had the same issue but using cuda 11.1 resolved it for me.

This is the exact pip command

pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html

Upvotes: 22

Marie Duc
Marie Duc

Reputation: 71

I am also using Cuda 10.2. I had the exact same error when upgrading torch and torchvision to the latest version (torch-1.8.0 and torchvision-0.9.0). Which version are you using?

I guess this is not the best solution but by downgrading to torch-1.7.1 and torchvision-0.8.2 it works just fine.

Upvotes: 7

Related Questions