CODEWITHSUNDEEP
pythonkerasdeep-learninggoogle-colaboratory
lycutter
lycutter

Reputation: 371

Buffered data was truncated after reaching the output size limit

When I use Colaboratory to run my NIN model, it occurs an error in the output of training process which tells "Buffered data was truncated after reaching the output size limit." in the 61th epoch. I have no idea about this crash.Is my code still running ? How can I solve this problem?

Here is some information about the output of my training process:

Epoch 57/200
391/391 [==============================] - 53s 135ms/step - loss: 0.8365 - acc: 0.7784 - val_loss: 0.9250 - val_acc: 0.7625
Epoch 58/200
 28/391 [=>............................] - ETA: 46s - loss: 0.8356 - acc: 0.7835391/391 [==============================] - 53s 136ms/step - loss: 0.8288 - acc: 0.7811 - val_loss: 0.8977 - val_acc: 0.7608
Epoch 59/200
326/391 [========================>.....] - ETA: 8s - loss: 0.8309 - acc: 0.7789391/391 [==============================] - 53s 136ms/step - loss: 0.8297 - acc: 0.7798 - val_loss: 0.9030 - val_acc: 0.7628
Epoch 60/200
391/391 [==============================] - 53s 134ms/step - loss: 0.8245 - acc: 0.7825 - val_loss: 0.8378 - val_acc: 0.7767
Epoch 61/200
 28/391 [=>............................] - ETA: 46s - loss: 0.8281 - acc: 0.7879390/391 [============================>.] - ETA: 0s - loss: 0.8177 - acc: 0.7851Buffered data was truncated after reaching the output size limit.

Upvotes: 25

Views: 25817

Answers (3)

Praveen kumar
Praveen kumar

Reputation: 768

Even if RAM | GPU | DISK on colab is free, this error still comes because there is a limited memory for displaying output of a cell on colab. Assuming the memory limit is around 2Mb to 5Mb when we run many epochs(148+) during training, it tends to fill that memory and hence the output is truncated because there is no more memory left free to display the buffered epochs. However, the machine keeps running in the background and the output is processed but it is not displayed because of the buffered limit. You will still get your desired output.

One solution is not to use verbose=1 (use 0 instead).

Upvotes: 33

nima farhadi
nima farhadi

Reputation: 698

it's not relating to hardware RAM or GPU capacity.

Keras framework has a limitation for showing output info in console.

when you see this message, your process is going on in background but you can't see it.

If you use tensorflow as backend, write a Tensorboard callback in your Keras to see detailed output of your network.

https://keras.io/callbacks/#tensorboard

Upvotes: 9

Farid.H
Farid.H

Reputation: 45

I think this error is for out of memory. Your RAM or GPU memory was full and could not process new data. you can do two works: 1. Decrease your batch size. 2. Save your model in for example 60th epoch and close current program and run new program and restore saved model and train model from 61 epoch to 120 epoch and save that and close program and repeat this work for your interested epoch

Upvotes: -12

Related Questions