CODEWITHSUNDEEP
deep-learningpytorchtransformer-modelhuggingface-transformers
Andrew Ishutin
Andrew Ishutin

Reputation: 99

BERT token importance measuring issue. Grad is none

I am trying to measure token importance for BERT via comparing token embedding grad value. So, to get the grad, I've copied the 2.8.0 forward of BertModel and changed it a bit:

huggingface transformers 2.8.0 BERT https://github.com/huggingface/transformers/blob/11c3257a18c4b5e1a3c1746eefd96f180358397b/src/transformers/modeling_bert.py

Code:

        embedding_output = self.embeddings(
            input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
        )
        embedding_output = embedding_output.requires_grad_(True) # my code
        encoder_outputs = self.encoder(
            embedding_output,
            attention_mask=extended_attention_mask,
            head_mask=head_mask,
            encoder_hidden_states=encoder_hidden_states,
            encoder_attention_mask=encoder_extended_attention_mask,
        )
        sequence_output = encoder_outputs[0]
        sequence_output.mean().backward() # my code
        assert(embedding_output.grad is not None) # my code

Colab link: https://colab.research.google.com/drive/1MggBUaDWAAZNuXbTDM11E8jvdMGEkuRD But it gives assertion error. I do not understand why and it seems to be a bug for me. Please, help!

Upvotes: 1

Views: 531

Answers (1)

Andrew Ishutin
Andrew Ishutin

Reputation: 99

I needed to add this line:

embedding_output = torch.tensor(embedding_output, requires_grad=True)

It seems, that I used .requires_grad_ method incorrectly.

Upvotes: 2

Related Questions