CODEWITHSUNDEEP
memory-leaksdeep-learningpytorchgpuconv-neural-network
Beginner
Beginner

Reputation: 749

Pytorch: How to know if GPU memory being utilised is actually needed or is there a memory leak

I have 3 Tesla V100(16 GB). I am doing transfer learning using efficeint net (63 Million parameters) on images of (512,512) with a batch size of 20. My GPU memory utilisation is below - enter image description here

As you can see, it has almost filled up all the 3 GPUs(almost 80%). My question is is there any theoretical way of calculating that the utilisation of GPU memory being shown is what is required by the model at a certain image and batch size or is there a memory leak in my GPU?

Upvotes: 1

Views: 576

Answers (1)

White
White

Reputation: 345

I'm not sure that is what you asked for but have you tried doing something like:

memory_usage = number_of_variables * memory_usage_per_variable.

So if you use torch.float32 tensors, and you have 125 000 variables sent to the GPU with .cuda(). Then you are using 4Gbytes of memory on your GPU. You can compare with how much memory you have available on your memory.

Another Sanity check would be to check memory usage on the GPU's per iteration your model, if it doubles then you have a memory leak.

Hope this helps.

Upvotes: 2

Related Questions