CODEWITHSUNDEEP
pythontensorflowmachine-learningtensorboard
chesschi
chesschi

Reputation: 708

Tensorflow: How to determine the value of clip_norm when using clip_by_norm or clip_by_global_norm?

I am trying to find out how to determine the value of clip_norm when using clip_by_norm or clip_by_global_norm with Tensorboard.

In Tensorflow, we can use the optimizer to compute_gradients to obtain the gradient and add it to the tf.summary.histogram. In Tensorboard, we can observe the range of the gradient in the DISTRIBUTIONS tab. Then we can determine the clip_value range when using clip_by_value.

On the other hand, how shall I calculate/display the norm or global norm in Tensorboard so that I can determine the value of clip_norm?

Upvotes: 2

Views: 1509

Answers (1)

dm0_
dm0_

Reputation: 2156

To view norm or global norm in Tensorboard you can manually calculate it.

For the norm (not global) you obtain gradient as usual using compute_gradients and compute gradient l2 norm using tf.norm. This gives you a scalar value. You can add it to Tensorboard using tf.summary.scalar.

For the global norm you can compute it using tf.global_norm and then add it to Tensorboard (also scalar summary).

EDIT:

Please note that tf.norm expects single tensor. So you need to compute norm and add to to Tensorboard for each gradients tensor. The same as you do when using tf.summary.histogram to view values.

Upvotes: 4

Related Questions