CODEWITHSUNDEEP
cuda
Ratacand
Ratacand

Reputation: 130

cuda-memcheck : internal error (7)

I was trying to debug my CUDA-C program and I found out cuda-memcheck could be helpful. My problem is that I can't use the tool, because everytime I run it, even on an "Hello World" program taken from the internet, I get this result:

========= CUDA-MEMCHECK
  *program output*
========= Internal error (7)
========= No CUDA-MEMCHECK results found

Here's also a screen of the bash: cuda-memcheck output

I found nothing on the Internet so I hope someone else here will be able to help me. Thanks anyone!

Upvotes: 4

Views: 2357

Answers (1)

Miles
Miles

Reputation: 885

It seems this is just a permissions issue, as first mentioned in the comments by Robert Crovella. Make sure you have permission to use the relevant /dev files, and can see /dev/nvidia-uvm-tools there.

For me, this issue came up inside a docker container. I then restarted the container with the privileged flag:

nvidia-docker run ... --privilieged ...

and this solved my issue entirely. See issue: https://github.com/NVIDIA/nvidia-docker/issues/454.

So either running memcheck with:

sudo cuda-memcheck ...

or trying to reinstall everything with correct permissions and running with sudo should solve this issue (as you found).

One thing that was evident for me was that cuda-memcheck ran quite quickly while I executed my program normally. This should not happen: the command can slow your kernels down over 10x, so if you notice the command running at normal speed to regular execution, that is a sign that cuda-memcheck can't even get inside your kernels to measure them.

Upvotes: 5

Related Questions