CODEWITHSUNDEEP
nlppytorchhuggingface-transformers
Sean
Sean

Reputation: 3450

How to get the logits for the T5 model when using the `generate` method for inference?

I'm currently using HuggingFace's T5 implementation for text generation purposes. More specifically, I'm using the T5ForConditionalGeneration to solve a text classification problem as generation.

The model's performance is overall very satisfactory after training, but what I am wondering is how I can get the logits for generation?

I'm currently performing inference as is suggested in the documentation via model.generate(**tokenizer_outputs), but this simply outputs the IDs themselves without anything else.

The reason why I want the logits is because I want to measure the model's confidence of generation. I'm not 100% certain if my approach is correct, but I'm thinking that if I can get the logit values of each generated token and average them, I could get the overall confidence score of the generated sequence.

Would anybody know how I could do this? Thanks.

Upvotes: 2

Views: 1722

Answers (1)

Sean
Sean

Reputation: 3450

I was struggling with this because I wasn't familiar with how the Transformers library works, but after looking at the source code all you have to do is set the arguments output_scores and return_dict_in_generate to True.

For more information, take a look at the method transformers.generation.utils.GenerationMixin.generate.

Upvotes: 3

Related Questions