CODEWITHSUNDEEP
pythongpucaffe
warped
warped

Reputation: 9482

Check failed: error == cudaSuccess (2 vs. 0) out of memory

I am trying to run a neural network with pycaffe on gpu.

This works when I call the script for the first time. When I run the same script for the second time, CUDA throws the error in the title.

Batch size is 1, image size at this moment is 243x322, the gpu has 8gb RAM.

I guess I am missing a command that resets the memory?

Thank you very much!

EDIT:

Maybe I should clarify a few things: I am running caffe on windows.

When i call the script with python script.py, the process terminates and the gpu memory gets freed, so this works.

With ipython, which I use for debugging, the GPU memory indeed does not get freed (after one pass, 6 of the 8 bg are in use, thanks for the nvidia-smi suggestion!)

So, what I am looking for is a command I can call from within pyhton, along the lines of:

run network

process image output

free gpu memory

Upvotes: 3

Views: 3519

Answers (2)

Harsh Wardhan
Harsh Wardhan

Reputation: 2148

Your GPU memory is not getting freed. This happens when the previous process is stopped but not terminated. See my answer here.

Upvotes: 3

malreddysid
malreddysid

Reputation: 1342

This happens when you run out of memory in the GPU. Are you sure you stopped the first script properly? Check the running processes on your system (ps -A in ubuntu) and see if the python script is still running. Kill it if it is. You should also check the memory being used in your GPU (nvidia-smi).

Upvotes: 2

Related Questions