CODEWITHSUNDEEP
cuda
user2100910
user2100910

Reputation: 327

Weird bug about cudaMemcpy

I got a very weird bug in my CUDA program. After a whole day of debugging, I have narrowed down the bug into this little piece of code, which simply copies data in arrays[0] to all other arrays[1~9]. All entries in arrays[0] are zero, so after the copying, everything in arrays[0~9] should be zero. However, the reality is that some memory is corrupted with random values. In my case, the some entries in arrays[8] are nonzero. I really appreciate it if someone can replicate my bug. Thank you so much!

Here's the code: (cudaDeviceSynchronize() added)

#include <iostream>
#include <assert.h>

#include <cuda.h>
#include <cuda_runtime.h>

using namespace std;

int main() {
  size_t count = 20 * 1000 * 1000;
  size_t size = count * sizeof(float);
  void *arrays[10];
  for (size_t i = 0; i < 10; i++) {
    assert(cudaMalloc(&arrays[i], size) == cudaSuccess);
    cudaDeviceSynchronize();
    assert(cudaMemset(arrays[i], 0, size) == cudaSuccess);
    cudaDeviceSynchronize();
  }
  void *host_array;
  assert(cudaMallocHost(&host_array, size) == cudaSuccess);
  for (size_t i = 1; i < 10; i++) {
    void *dst_array = arrays[i];
    void *src_array = arrays[0];
    assert(cudaMemcpy(dst_array, src_array, size, cudaMemcpyDeviceToDevice) == cudaSuccess);
    cudaDeviceSynchronize();
    cout << "copied" << i << endl;
    for (size_t j = 0; j < 10; j++) {
      void *array = arrays[j];
      assert(cudaMemcpy(host_array, array, size, cudaMemcpyDeviceToHost) == cudaSuccess);
      cudaDeviceSynchronize();
      float *floats = reinterpret_cast<float *>(host_array);
      float sum = 0.0;
      for (size_t k = 0; k < count; k++) {
        sum += floats[k];
      }
      cout << "sum" << j << "=" << sum << endl;
    }
  }
}

I compiled it using:

nvcc test.cu -lrt -lcudart -o test -g -O3

The output I got is:

copied1
sum0=0
sum1=0
sum2=0
sum3=0
sum4=0
sum5=0
sum6=0
sum7=0
sum8=4.23087e+20
sum9=0
copied2
sum0=0
sum1=0
sum2=0
sum3=0
sum4=0
sum5=0
sum6=0
sum7=0
sum8=inf
sum9=0
copied3
sum0=0
sum1=0
sum2=0
sum3=0
sum4=0
sum5=0
sum6=0
sum7=0
sum8=inf
sum9=0
copied4
sum0=0
sum1=0
sum2=0
sum3=0
sum4=0
sum5=0
sum6=0
sum7=0
sum8=inf
sum9=0
copied5
sum0=0
sum1=0
sum2=0
sum3=0
sum4=0
sum5=0
sum6=0
sum7=0
sum8=inf
sum9=0
copied6
sum0=0
sum1=0
sum2=0
sum3=0
sum4=0
sum5=0
sum6=0
sum7=0
sum8=-1.08991e+38
sum9=0
copied7
sum0=0
sum1=0
sum2=0
sum3=0
sum4=0
sum5=0
sum6=0
sum7=0
sum8=-nan
sum9=0
copied8
sum0=0
sum1=0
sum2=0
sum3=0
sum4=0
sum5=0
sum6=0
sum7=0
sum8=0.0234719
sum9=0
copied9
sum0=0
sum1=0
sum2=0
sum3=0
sum4=0
sum5=0
sum6=0
sum7=0
sum8=2.51686e+17
sum9=0

Thanks, Cui

Upvotes: 1

Views: 223

Answers (1)

user2100910
user2100910

Reputation: 327

My problem is solved by rolling back to an older CUDA driver version. It seems that we upgraded our CUDA toolkit to 7.0 a few days ago, and a new CUDA driver of version 346.46 was installed. I know it makes no sense, but this driver is causing the problem. The problem is solved when I reinstalled our previous driver.

Thank you for all your efforts of helping me. That is definitely the hardest bug I have ever seen!

Thanks,

Cui

Upvotes: 1

Related Questions