CODEWITHSUNDEEP
pythonpytorchsoftmax
JHPark
JHPark

Reputation: 31

do I have to add softmax in def forward when I use torch.nn.CrossEntropyLoss

https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html

When I read the contents above, I understood that torch.nn.CrossEntropy already computes exp score of the last layer. So I thought the forward function doesn't have to include softmax. For example, return self.fc(x) rather than return nn.softmax(self.fc(x)). However, I'm confused, for I've seen several implementations of ConvNet Classifiers that use both ways (they return with or without softmax while both use cross entropy loss).

Do this issues affect the performance of Classifier?? Which way is correct?

Upvotes: 2

Views: 1539

Answers (1)

Poe Dator
Poe Dator

Reputation: 4903

JHPark,

You are correct - with torch.nn.CrossEntropyLoss there is no need to include softmax layer. If one does include softmax it will still lead to proper classification result, since softmax does not change which element has max score. However, if applied twice, it may distort relative levels of the outputs, making gradients less strong and potentially slowing training a bit.

Upvotes: 2

Related Questions