CODEWITHSUNDEEP
pythontensorflowkerasconv-neural-network
Tom Halmos
Tom Halmos

Reputation: 113

How to solve the 'BiasGrad requires tensor size <= int32 max' InvalidArgumentError in TensorFlow?

I'm trying to train a convolutional neural network using Keras/Tensorflow. My model compiles correctly, but as soon as training begins the following error is returned:

Using TensorFlow backend.

Epoch 1/3

Traceback (most recent call last):
  File "./main.py", line 17, in <module>
    history = CNN.fit(TrainImages, TrainMasks, epochs = 3)

  File "/home/tomhalmos/.local/lib/python3.6/site-packages/keras/engine/training.py", line 1239, in fit
    validation_freq=validation_freq)

  File "/home/tomhalmos/.local/lib/python3.6/site-packages/keras/engine/training_arrays.py", line 196, in fit_loop
    outs = fit_function(ins_batch)

  File "/home/tomhalmos/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/backend.py", line 3727, in __call__
    outputs = self._graph_fn(*converted_inputs)

  File "/home/tomhalmos/.local/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 1551, in __call__
    return self._call_impl(args, kwargs)

  File "/home/tomhalmos/.local/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 1591, in _call_impl
    return self._call_flat(args, self.captured_inputs, cancellation_manager)

  File "/home/tomhalmos/.local/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 1692, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))

  File "/home/tomhalmos/.local/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 545, in call
    ctx=ctx)

  File "/home/tomhalmos/.local/lib/python3.6/site-packages/tensorflow_core/python/eager/execute.py", line 67, in quick_execute
    six.raise_from(core._status_to_exception(e.code, message), None)

  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.**InvalidArgumentError:  BiasGrad requires tensor size <= int32 max**
         [[node gradients/conv2d_22/BiasAdd_grad/BiasAddGrad (defined at /home/tomhalmos/.local/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3009) ]] [Op:__inference_keras_scratch_graph_5496]

Function call stack:
keras_scratch_graph

Happy to provide any further details if the above is not sufficient.

Upvotes: 5

Views: 1843

Answers (2)

Johnson Chen
Johnson Chen

Reputation: 131

I change my job on GPU and it's work well.

Upvotes: 1

Robert Lugg
Robert Lugg

Reputation: 1192

The bounds checking is on the number of elements in a tensor. The size is limited to 2.147 billion values (int32).

Take your image size (h x v) times the sample batch size. Multiply that by the number of channels in your operation (such as Conv2D). The place where you get a count larger than 2.1e9 is the guilty operation. There is no solution that I can see other than reducing one of those numbers.

Upvotes: 4

Related Questions