CODEWITHSUNDEEP
tensorflow
SuaveSouris
SuaveSouris

Reputation: 1342

Tensorflow softmax_cross_entropy_with_logits asks for unscaled log probabilities

I have noticed that tf.nn.softmax_cross_entropy_with_logits asks for "unscaled log probabilities" for the logits argument. However, nowhere have I seen anyone suggest performing a log operation on their NN predictions before submission to this function. Am I missing something here?

Upvotes: 3

Views: 1856

Answers (2)

Hatem Zegnani
Hatem Zegnani

Reputation: 1

for historical reasons, the output of last linear layer say:

Z2=np.dot(W2,A1)+b2 ,

this output is called "unscaled log probabilities" thus the softmax operation will normalize them --> giving the "scaled log probabilities" and apply the exponential function --> giving the "scaled probabilities" --> on which it applies the cross-entropy loss

see: http://cs231n.github.io/neural-networks-2/

Upvotes: 0

Alexandre Passos
Alexandre Passos

Reputation: 5206

You shouldn't perform a log operation. You shouldn't perform anything, actually :-). What this comment is (arguably poorly) trying to say is that each logit is an unrestricted real number (negative or positive, as big or as small as you want). The softmax cross entropy function will then (conceptually) apply the softmax operation (exponentiate, to turn unrestricted real numbers into positive numbers, and then normalize to make them sum to 1) and compute the cross-entropy.

So, tl;dr, feed the outputs of your last linear layer without any normalization or transfer function to this and you won't be wrong.

Upvotes: 8

Related Questions