CODEWITHSUNDEEP
pythontensorflow
Someone
Someone

Reputation: 213

Tensorflow GAN discriminator loss NaN since negativ discriminator output

In my implementation of a GAN network the output of the discriminator is something like 2.05145e+07 which leads to 1 - disc_output -> 1-2.05145e+07=-2.05145e+07 (a negativ number) therefore log(1-2.05145e+07) leads to NaN.

I am not the first one with this kind of problem. One solution is to only allow positive values inside the log like done here.

Does anyone knows any better solution to this?
maybe some different loss function ?

Upvotes: 1

Views: 3469

Answers (2)

Joel Shor
Joel Shor

Reputation: 66

There are standard techniques to avoid log numerical instability. For example, what you often care about is the loss (which is a function of the log), not the log value itself. For instance, with logistic loss:

For brevity, let x = logits, z = labels. The logistic loss is

z * -log(sigmoid(x)) + (1 - z) * -log(1 - sigmoid(x))

= max(x, 0) - x * z + log(1 + exp(-abs(x)))

These tricks are already implemented in standard tensorflow losses (like tf.losses.sigmoid_cross_entropy). Note that the naive solution of taking a max or a min inside of the log is not a good solution, since there aren't meaningful gradients in the saturated regions: for instance, d/dx[max(x, 0)] = 0 for x < 0, which means there won't be gradients in the saturated region.

TensorFlow has GAN support with tf.contrib.gan. These losses already implement all of the standard numerical stability tricks, and an avoid you having to recreate the wheel.

tfgan = tf.contrib.gan

tfgan.losses.minimax_discriminator_loss(...)

See https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/gan for more details.

Upvotes: 1

eaksan
eaksan

Reputation: 575

Because discriminator returns a probability value, its output must be between 0 and 1. Try using sigmoid ( https://www.tensorflow.org/api_docs/python/tf/sigmoid) before using discriminator outputs.

Additionally, as others did, I suggest using tf.log(tf.maximum(x, 1e-9)) in case of a numerical instability.

Upvotes: 2

Related Questions