CODEWITHSUNDEEP
pythonnlppytorchbert-language-modelhuggingface-transformers
Ishan Dutta
Ishan Dutta

Reputation: 957

Python: BERT Error - Some weights of the model checkpoint at were not used when initializing BertModel

I am creating an entity extraction model in PyTorch using bert-base-uncased but when I try to run the model I get this error:

Some weights of the model checkpoint at D:\Transformers\bert-entity-extraction\input\bert-base-uncased_L-12_H-768_A-12 were not used when initializing BertModel:    
['cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight',   'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.bias',  
 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.weight',  
 'cls.predictions.bias']  
    - This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
    - This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

I have downloaded the bert model from here and the additional files from here.

Following is the code for my model:

import config
import torch
import transformers
import torch.nn as nn

def loss_fn(output, target, mask, num_labels):

    lfn = nn.CrossEntropyLoss()
    active_loss = mask.view(-1) == 1
    active_logits = output.view(-1, num_labels)
    active_labels = torch.where(
        active_loss,
        target.view(-1),
        torch.tensor(lfn.ignore_index).type_as(target)
    )
    loss = lfn(active_logits, active_labels)
    return loss

class EntityModel(nn.Module):
    def __init__(self, num_tag, num_pos):
        super(EntityModel, self).__init__()

        self.num_tag = num_tag
        self.num_pos = num_pos
        self.bert = transformers.BertModel.from_pretrained(config.BASE_MODEL_PATH)
        self.bert_drop_1 = nn.Dropout(p = 0.3)
        self.bert_drop_2 = nn.Dropout(p = 0.3)
        self.out_tag = nn.Linear(768, self.num_tag)
        self.out_pos = nn.Linear(768, self.num_pos)

    def forward(self, ids, mask, token_type_ids, target_pos, target_tag):
        o1, _ = self.bert(ids, 
                          attention_mask = mask,
                          token_type_ids = token_type_ids)

        bo_tag = self.bert_drop_1(o1)
        bo_pos = self.bert_drop_2(o1)

        tag = self.out_tag(bo_tag)
        pos = self.out_pos(bo_pos)

        loss_tag = loss_fn(tag, target_tag, mask, self.num_tag)
        loss_pos = loss_fn(pos, target_pos, mask, self.num_pos)

        loss = (loss_tag + loss_pos) / 2

        return tag, pos, loss 

print("model.py run success!")

Upvotes: 17

Views: 42107

Answers (3)

Jatin Gupta
Jatin Gupta

Reputation: 1

I was using AutoModel and saving the model than AutoModelForSequenceClassification or specific to load from_pretrained and save_pretrained

Upvotes: 0

Micheal Bee
Micheal Bee

Reputation: 628

As R. Marolahy suggests, if you don't want to see this every time, I know I don't, add the following:

from transformers import logging
logging.set_verbosity_error()

Upvotes: 23

R. Marolahy
R. Marolahy

Reputation: 1578

As @cronoik and mentionned here , this is not an error but this warning means that during your training, you're not using the pooler in order to compute the loss. So if that is the case there's no need to worry about it.

You can set this warning by doing:

from transformers import logging

logging.set_verbosity_warning()

Upvotes: 10

Related Questions