cudaError_t 1 : "__global__ function call is not configured" returned from 'cublasCreate(&handle_)'

Question

I run ASR experiment using Kaldi on SGE cluster consisting of two workstation with TITAN XP. And randomly I meet the following problem:

ERROR (nnet3-train[5.2.62~4-a2342]:FinalizeActiveGpu():cu-device.cc:217) cudaError_t 1 : "__global__ function call is not configured" returned from 'cublasCreate(&handle_)'

I guess something is wrong with GPU driver or hardware.

Could you please offer some help?

And here is the complete log

Answer 1

I had similar issue in running darknet in one of the TX2

with reference to

https://blog.csdn.net/JIEJINQUANIL/article/details/103091537

enter the root by

sudo su

Then source the catkin_ws

Then launch the darkent.

Then can run.

Here is my result

Hope you can solve it by similar method