Tensorflow CNN model get error " NaN loss during training."

Question

I am a newer to tensorflow. When I training the data with cnn ,sometimes "NaN loss during training" is happened at the first batch.

> loss function is L1-norm
> 
> GradientDescentOptimizer is used.

"bach_size" ,"learning_rate" are adjust, even learning_rate = 0 is used.

What I really confused is when all parameters fixed, the result is sometimes can run normally ,sometimes get the error "Nan" at first batch. I want to know how it happens? what factors lead to the result?

Answer 1

• Check for NANs in your data
• RELU activation gives sometimes NANs