CODEWITHSUNDEEP
pytorch-lightning
pseudo_teetotaler
pseudo_teetotaler

Reputation: 1575

Training Data Split across GPUs in DDP Pytorch Lightning

Goal: Train a model in Distriubted Data Parallel(DDP) setting using Pytorch Lightning Framework

Questions:

  1. Training Data Partition: How is data partition across separate GPUs is handled with Pytorch Lightning? Am I supposed to manually partition the data or Pytorch lightning will take care of that?

  2. Loss Averaging: Do I have to aggregate the losses myself or Pytorch Lightning is going automatically do that?

I have been spending time with the code-base of pytorch lightning, looking for how DDP sync is handled, but unable to find that exact code. Would appreciate a clarification on this.

Upvotes: 1

Views: 705

Answers (1)

Ari
Ari

Reputation: 583

Lightning handles both of these scenarios for you out of the for you but it can be overridden. The code for this can be found in the official github here.

Upvotes: 3

Related Questions