CODEWITHSUNDEEP
nlppytorchhuggingface-transformers
Dilrukshi Perera
Dilrukshi Perera

Reputation: 997

How to calculate perplexity for a language model using Pytorch

I'm fine-tuning GPT-2 model for language generation task using huggingface Transformers library-pytorch, and I need to calculate an evaluation score(perplexity) for the fine-tuned model. But I'm not sure how that can be done using loss. I would like to know how perplexity can be calculated for the model with sum_loss or mean loss or any other suggestions are also welcome. Any help is appriciated.

Edit:

outputs = model(article_tens, labels=article_tens)

        loss, prediction_scores = outputs[:2]                        
        loss.backward()
        sum_loss = sum_loss + loss.detach().data

given above is how I calculate loss for each batch of data for the fine-tuning task.

sum loss 1529.43408203125
loss 4.632936000823975
prediction_scores tensor([[[-11.2291,  -9.2614, -11.8575,  ..., -18.1927, -17.7286, -11.9215],
         [-67.2786, -63.5928, -70.7110,  ..., -75.3516, -73.8672, -67.6637],
         [-81.1397, -80.0295, -82.9357,  ..., -83.7913, -85.7201, -78.9877],
         ...,
         [-85.3213, -82.5135, -86.5459,  ..., -90.9160, -90.4393, -82.3141],
         [-44.2260, -43.1702, -49.2296,  ..., -58.9839, -55.2058, -42.3312],
         [-63.2842, -59.7334, -61.8444,  ..., -75.0798, -75.7507, -54.0160]]],
       device='cuda:0', grad_fn=<UnsafeViewBackward>)

The above given is when loss is printed for a batch only

Upvotes: 7

Views: 12926

Answers (1)

Michael Jungo
Michael Jungo

Reputation: 32972

As shown in Wikipedia - Perplexity of a probability model, the formula to calculate the perplexity of a probability model is:

Perplexity Formula

The exponent is the cross-entropy. While logarithm base 2 (b = 2) is traditionally used in cross-entropy, deep learning frameworks such as PyTorch use the natural logarithm (b = e).

Therefore, to get the perplexity from the cross-entropy loss, you only need to apply torch.exp to the loss.

perplexity = torch.exp(loss)

The mean loss is used in this case (the 1 / N part of the exponent) and if you were to use the sum of the losses instead of the mean, the perplexity would get out of hand (exceedingly large), which can easily surpass the maximum floating point number, resulting in infinity.

Upvotes: 11

Related Questions