Delete and Reinitialize pertained BERT weights / parameters

Question

I tried to fine-tune BERT for a classification downstream task.

Now I loaded the model again and I run into the following warning:

Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.weight', 'cls.predictions.bias', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight']

• This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
• This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

Screen Shot

I already deleted and reinstalled transformers==4.6.0 but nothing helped. I thought maybe through the parameter "force_download=True" it might get the original weights back but nothing helped.

Shall I continue and ignore the warning? Is there a way to delete the model checkpoints such when the model is downloaded the weights are fixed again?

Answer 1

I have kind of "solved" the problem or at least I found a solution:

• One Conda environment for downstream task classification: conda install -c conda-forge transformers
• One Conda environment for just getting the embeddings: conda install -c conda-forge/label/cf202003 transformers

Answer 2

As long as you're fine-tuning a model for a downstream task this warning can be ignored. The idea is that the [CLS] token weights from the pretrained model aren't going to be useful for downstream tasks and need to be fine-tuned.

Huggingface randomly initializes them because you're using bert-base-cased which is a BertForPretraing model and you're created a BertModel from it. The warning is to ensure that you understand the difference of directly using the pretrained model directly or if you're planning on finetuning them for a different task.

On that note if you plan working on a classification task I'd recommend using their BertForSequenceClassification class instead.

TL;DR you can ignore it as long as you're finetuning.