CAFFE: Cuda Error "(8 vs. 0) invalid device function" when using GPU (GeForce GTX 970)?

Question

I'm trying to run the CNN Network "CAFFE" on a GTX 970. But I get the error mentioned in the title.

Can someone help?

I posted the issue with more details on the caffe group, but received no hints/answers!

https://groups.google.com/forum/#!topic/caffe-users/sVOfE0qhf_M

update 1

In my Makefile.config, I added -gencode arch=compute_52,code=compute_52

CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \
        -gencode arch=compute_20,code=sm_21 \
        -gencode arch=compute_30,code=sm_30 \
        -gencode arch=compute_35,code=sm_35 \
        -gencode arch=compute_35,code=sm_35 \
        -gencode arch=compute_50,code=compute_50 \
        -gencode arch=compute_52,code=compute_52     

But when I try to make, it returns:

$make
NVCC src/caffe/layers/cudnn_sigmoid_layer.cu
nvcc fatal   : Unsupported gpu architecture 'compute_52'
Makefile:531: recipe for target '.build_release/cuda/src/caffe/layers/cudnn_sigmoid_layer.o' failed
make: *** [.build_release/cuda/src/caffe/layers/cudnn_sigmoid_layer.o] Error 1

update 2

The NCC version is:

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2014 NVIDIA Corporation
Built on Thu_Jul_17_21:41:27_CDT_2014
Cuda compilation tools, release 6.5, V6.5.12

update 3

I'm using CUDA 6.5 with the 346.96 driver

libcuda1-346
/.
/usr
/usr/lib
/usr/lib/i386-linux-gnu
/usr/lib/i386-linux-gnu/libcuda.so
/usr/lib/i386-linux-gnu/libcuda.so.1
/usr/lib/i386-linux-gnu/libcuda.so.346.96
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/libcuda.so
/usr/lib/x86_64-linux-gnu/libcuda.so.1
/usr/lib/x86_64-linux-gnu/libcuda.so.346.96
/usr/share
/usr/share/doc
/usr/share/doc/libcuda1-346
/usr/share/doc/libcuda1-346/changelog.Debian.gz
/usr/share/doc/libcuda1-346/copyright


libcudart6.5
/.
/usr
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/libcudart.so.6.5
/usr/lib/x86_64-linux-gnu/libcudart.so.6.5.14
/usr/share
/usr/share/doc
/usr/share/doc/libcudart6.5
/usr/share/doc/libcudart6.5/changelog.Debian.gz
/usr/share/doc/libcudart6.5/copyright
/usr/share/lintian
/usr/share/lintian/overrides
/usr/share/lintian/overrides/libcudart6.5

update 4

I found here (page is in German) that Ubuntu 15.04 only supports NVidia GPUs until the 800-series. What I'm doing right now is update Ubuntu to 15.10, where the NVidia driver version "nvidia-352" is available that is supposed to support a Geforce GTX 970.

I'll post the results here.

update 5

It's updated to Ubuntu 15.10. This comes with GCC 5.2, but CAFFE only accepts versions below 4.9. I installed g++ 4.8 parallel to the existing one I've set the compiler to "g++-4.8" in cmake...

update 6

Now the error reads

[  1%] Built target proto
[  1%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/util/cuda_compile_generated_math_functions.cu.o
nvcc fatal   : Unsupported gpu architecture 'compute_52'
CMake Error at cuda_compile_generated_math_functions.cu.o.cmake:206 (message):
  Error generating
  /home/art/Downloads/caffe-master-build/src/caffe/CMakeFiles/cuda_compile.dir/util/./cuda_compile_generated_math_functions.cu.o


src/caffe/CMakeFiles/caffe.dir/build.make:375: recipe for target 'src/caffe/CMakeFiles/cuda_compile.dir/util/cuda_compile_generated_math_functions.cu.o' failed
make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/util/cuda_compile_generated_math_functions.cu.o] Error 1
CMakeFiles/Makefile2:218: recipe for target 'src/caffe/CMakeFiles/caffe.dir/all' failed
make[1]: *** [src/caffe/CMakeFiles/caffe.dir/all] Error 2
Makefile:116: recipe for target 'all' failed
make: *** [all] Error 2

update 7

My next attempt:

1) Install Ubuntu 15.04

2) Install CUDA 7.5 directly from nVidia

3) Install cuDNN 7.0

4) install CAFFE

that finally worked!!

Answer 1

Whenever the CUDA runtime API returns "Invalid Device Function", it means you are using code which wasn't built for the architecture you are trying to run it on (and doesn't have a JIT path).

You probably need to check your CAFFE Makefile.config to make sure it is set for the correct architecture, by making sure that CUDA_ARCH includes -gencode arch=compute_52,code=compute_52.

Because you are using a compute capability 52 device, you must use CUDA 7 or newer with your GPU. The CUDA 6.5 toolkit you are using cannot generate code for your GPU, and the CUDA 6.5 driver cannot JIT recompile lower compute capability code into something which will run on your GPU. This is a not negotiable requirement for your hardware.