CODEWITHSUNDEEP
tensorflow
Stefan Falk
Stefan Falk

Reputation: 25447

Getting 'undefined symbol: cudnnCreate'

I am trying to run a quite simple Tensorflow graph, but as I run the script I am getting the following output:

/usr/bin/python3.5 /media/Data/workspaces/python/tf_playground/play/cnn.py
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:119] Couldn't open CUDA library libcudnn.so. LD_LIBRARY_PATH: /opt/pycharm/pycharm-community-2016.3.2/bin:
I tensorflow/stream_executor/cuda/cuda_dnn.cc:3459] Unable to load cuDNN DSO
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so locally
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties: 
name: GeForce GTX 1070
major: 6 minor: 1 memoryClockRate (GHz) 1.683
pciBusID 0000:01:00.0
Total memory: 7.92GiB
Free memory: 222.31MiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0:   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0)
F tensorflow/stream_executor/cuda/cuda_dnn.cc:221] Check failed: s.ok() could not find cudnnCreate in cudnn DSO; dlerror: /usr/local/lib/python3.5/dist-packages/tensorflow/python/_pywrap_tensorflow.so: undefined symbol: cudnnCreate

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

Any idea what the problem is here?

The strange thing is that I am able to run the MNIST softmax example without any errors.

This is the script which I am getting the error from:

import json

import requests
import tensorflow as tf
import numpy as np


class MyCNN(object):

    def __init__(self, sequence_length, num_classes, embedding_size, filter_sizes):

        self.input_x = tf.placeholder(tf.float32, [sequence_length, embedding_size], name="input_x")
        self.input_y = tf.placeholder(tf.float32, [None, num_classes], name="input_y")
        self.dropout_keep_prob = tf.placeholder(tf.float32, name="dropout_keep_prob")

        input_X = tf.reshape(self.input_x, [1, sequence_length, embedding_size, 1])

        pooled_outputs = []
        num_filters = len(filter_sizes)

        for i, filter_size in enumerate(filter_sizes):

            filter_shape = [filter_size, embedding_size, 1, num_filters]

            F = tf.Variable(tf.truncated_normal(filter_shape, stddev=0.1), name="F")
            b = tf.Variable(tf.constant(0.1, shape=[num_filters]), name="b")

            conv = tf.nn.conv2d(
                input_X,
                F,
                strides=[1, 1, 1, 1],
                padding="VALID",
                name="conv")

            # Apply nonlinearity
            h = tf.nn.relu(tf.nn.bias_add(conv, b), name="relu")

            # Maxpooling over the outputs
            pooled = tf.nn.max_pool(
                h,
                ksize=[1, sequence_length - filter_size + 1, 1, 1],
                strides=[1, 1, 1, 1],
                padding='VALID',
                name="pool")

            pooled_outputs.append(pooled)

        self.h_pool = tf.concat(3, pooled_outputs)

if __name__ == "__main__":

    headers = {
        "Content-Type": "application/json"
    }
    request = requests.post("http://localhost:8080/ema-server/w2v/getWordVectors",
                            data=json.dumps(["I", "really", "love", "to", "eat", "a", "lot", "of", "sushi!"]),
                            headers=headers)

    words = json.loads(request.text)

    X = []
    for word in words:
        if word is None: X.append([0] * 300); continue
        X.append(word)

    while len(X) < 50: X.append([0] * 300)

    X = np.asmatrix(X)
    X = np.reshape(X, [1, 50, 300, 1])

    cnn = MyCNN(50, 2, 300, [3])

    sess = tf.Session()

    sess.run(tf.global_variables_initializer())
    sess.run(cnn.h_pool, feed_dict={cnn.input_x: X})

    print("All done.")

Update: I followed these instructions to install CudNN but I am still getting the same error ..

Upvotes: 1

Views: 8082

Answers (5)

Abhishek
Abhishek

Reputation: 131

There are two ways to solve this problem (I faced it too)-
1)Either cudnn is not installed properly, to check this go to Path- /usr/local/cuda/lib64 if you are unable to find libcudnn.so then this working properly. Also check .bashrc file whether LD_LIBRARY_PATH is set to the above path. If its not set it.
2)The problem may be with Tensorflow version (occurs in most of the cases). Try to update tensorflow from tensorflow website. I faced the same issue and this was the reason.

Upvotes: 0

Sarvagya Gupta
Sarvagya Gupta

Reputation: 907

I solved it bu updating to tensorflow-gpu to 1.4. I was getting this in a conda environment. Updating tf-gpu solved the error.

Upvotes: 0

Simon Su
Simon Su

Reputation: 111

All above answer is right, that you need to set the enviroment right.

But if you install the wrong version of cudnn (even if it is the newer version), you will get this warning. You must install the right version of cuda and cudnn. You may follow the instructions on tensorflow.org, no TF re-compile is needed, just cp cudnn files to /user/local/cuda/*, and try again:

https://www.tensorflow.org/versions/r0.12/get_started/os_setup#optional_install_cuda_gpus_on_linux

enter image description here

Upvotes: 0

Sonu
Sonu

Reputation: 732

I solved the same problem by setting the export path for CUDA and CUDA/lib.

Upvotes: 1

Karthik N G
Karthik N G

Reputation: 2321

I got this error in windows environment

Check failed: s.ok() could not find cudnnCreate in cudnn DSO; dlerror: cudnnCreate not found

Assumption: You have already downloaded tensorflow with GPU support and have installed CUDA toolkit from NVIDIA from https://developer.nvidia.com/cuda-downloads

I followed the below steps to resolve it.

  1. Download cuDNN libraries from NVIDIA from the URL. https://developer.nvidia.com/cudnn
  2. Create a free account, answer all the questions while creating the account.

  3. You should be able to download the zip file for respective platform. See below for the options enter image description here

  4. Since I am using Windows 8.1. I downloaded library for Windows 10. It is zip file which contains three folders -> bin, inlcude and lib folders.

  5. Extract the files to a folder
  6. With default CUDA toolkit installation would be something in the folder -> C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA
  7. Copy the files from step 6 to the location given in the step 6. All files from bin go into bin folder in step 6 and so on.

These steps solved the error given above. Hopefully, this is helpful for someone.

Upvotes: 1

Related Questions