CODEWITHSUNDEEP
tensorflow
ebramos
ebramos

Reputation: 103

Is there a way to run each segment of a batch on different GPU?

So I am very new to Tensorflow and GPUs and I was wondering if I can feed different segments of my batch to different GPUs and aggregate the result at the end. What I mean is that, let's say that batch size in each epoch of my training is 1600 and I have 4 GPUs. Can I feed batches of size 400 to each GPU during each epoch of training and then aggregate the result?

Upvotes: 0

Views: 113

Answers (1)

Ujjwal
Ujjwal

Reputation: 208

You can do that. You will have to perform multi-gpu training.

Though in TensorFlow you can do a tower-based design where you collect and aggregate the gradients from each tower before backpropagation, it is not so simple and efficient.

You should use horovod which is easy and efficient.

Upvotes: 1

Related Questions