CODEWITHSUNDEEP
cuda
John Mansell
John Mansell

Reputation: 644

CUDA memory consumed when leaving function

I was running out of memory on my gtx1060, so I started checking available memory. I have one function that does a lot of computations on the GPU with thrust vectors, and then a second function which processes the results of function_1.

I've checked memory at the end of function_1, and immediately after returning from function_1, and the free memory is vastly different.

void debug_check_gpu_memory()
    {
        float free_m, total_m, used_m;
        size_t free_t, total_t;

        cudaMemGetInfo(&free_t, &total_t);

        free_m  =(uint) free_t/1048576.0 ;
        total_m =(uint)total_t/1048576.0;
        used_m  = total_m - free_m;


        cout << "     Memory :: " << endl;
        cout << "          Free = " << free_m << endl;
        cout << "          Used = " << used_m << endl;
        cout << "         Total = " << total_m << endl;
        cout << "-----------------------------------" << endl;
        cout << "\n\n";

    }

void function_1()
    {
        ...
        ...
        debug_check_gpu_memory();
           // Memory ::
           //    Free  =  3536.88
           //    Used  = -1373.69
           //    Total =  1979.19
        return;
    }

void function_2()
    {
        debug_check_gpu_memory();
           // Memory ::
           //    Free  =  728.87
           //    Used  = 1250.31
           //    Total = 1979.19
    }

main()
    {
           function_1;

           debug_check_gpu_memory();
           // Memory ::
           //    Free  =  728.87
           //    Used  = 1250.31
           //    Total = 1979.19

           function_2;

    }

Question

  1. Why am I seeing a negative number for "Used" memory in the first function? I check the memory throughout the first function, and its always negative.

  2. Why is there a different amount of memory used between the end of function_1 and checking again in main, or in function_2 ?

System:

Linux - Fedora 29 64 bit
NVIDIA-SMI 430.14
Cuda version : 10.2
Memory 6075 MiB
-- I also don't understand why I only see 1979 mb of usable memory when nvidia-smi only shows me using ~500 mb outside my program.

Upvotes: 1

Views: 204

Answers (1)

John Mansell
John Mansell

Reputation: 644

Thanks to @Robert_Crovella 's comment, I removed the offending type cast, and the problem went away. This wasn't a CUDA problem, this was a type casting problem.

I had been using the code found in this post on the Cuda forums, which turned out not to be 64 bit safe. I tried using the get_free_memory() function found here instead, and it worked quite well.

Upvotes: 1

Related Questions