CODEWITHSUNDEEP
pythontensorflow
Petr Petrov
Petr Petrov

Reputation: 4442

Tensorflow: on what does the batch_size depend?

I new on tensorflow and I try to understand what size should be batch.

Shape of my data (119396, 12955). How can I choose best batch_size to my data? And what dependence batch_size from data shape or using algorithm?

Upvotes: 3

Views: 1546

Answers (1)

ibarrond
ibarrond

Reputation: 7591

The batch size is the number of input data values that you are introducing at once in the model. It is very important while training, and secondary when testing. For a standard Machine Learning/Deep Learning algorithm, choosing a batch size will have an impact on several aspects:

  • The bigger the batch size, the more data you will feed at once in a model. Thus, RAM memory consumption will be almost linear with batch size, and there will always be a limit based on your system specs and the size of your model above which your model will overflow.
  • The bigger the batch size, the faster you will loop over your dataset N times to perform training.
  • A biggerbatch size will slow down your model training speed, meaning that it will take longer for your model to get one single update since that update depends on more data.
  • A biggerbatch size will have more data to average towards the next update of the model, hence training should be smoother: smoother training/test accuracy curves.

Note that the size of the data is only related to the batch size in the sense that the bigger the data, the smaller the maximum batch size becomes (limit set by RAM). The size of the model also has a similar relation.

In practice, you should follow "in powers of 2 and the larger the better, provided that the batch fits into your (GPU) memory". For more in-depth details, check https://stackoverflow.com/a/46655895/9670056.

Upvotes: 7

Related Questions