CODEWITHSUNDEEP
machine-learningpytorchgpudistributed-training
pythonHua
pythonHua

Reputation: 91

How is micro-batch-size influencing the throughput per GPU?

I am testing how is micro-batch-size influencing the throughput per GPU with a constant global-batch-size on Megatron-LM.

I have done several tests with a 400M transformer based model on 2 A40 GPUS, and only use data parallelism. Here are some training Arguments

enter image description here

With different test I only change the micro-batch-size , trained on 100 iterations with seq_len =1024 and global-batch-size =24 . Here are some result with different micro-batch-size enter image description here

I print the log every 5 iterations with the megatron-LM native functionality and compute the averaged throughput per GPU through all iterations. For each Iteration , the total computational complexity is the same , but throughput per GPU increases as the micro-batch-size increases. I know that may related to the GPU cache load or arithmetic intensity but not quite clear. Can anyone provide some in-depth explanations?

Upvotes: 0

Views: 87

Answers (0)

Related Questions