CODEWITHSUNDEEP
pythontensorflowmachine-learningpytorchhuggingface-transformers
Adirtha1704
Adirtha1704

Reputation: 359

Loading a converted pytorch model in huggingface transformers properly

I converted a pre-trained tf model to pytorch using the following function.

def convert_tf_checkpoint_to_pytorch(*, tf_checkpoint_path, albert_config_file, pytorch_dump_path):
    # Initialise PyTorch model
    config = AlbertConfig.from_json_file(albert_config_file)
    print("Building PyTorch model from configuration: {}".format(str(config)))
    model = AlbertForPreTraining(config)

    # Load weights from tf checkpoint
    load_tf_weights_in_albert(model, config, tf_checkpoint_path)

    # Save pytorch-model
    print("Save PyTorch model to {}".format(pytorch_dump_path))
    torch.save(model.state_dict(), pytorch_dump_path)

I am loading the converted model and encoding sentences in the following way:

def vectorize_sentence(text):
    albert_tokenizer = AlbertTokenizer.from_pretrained("albert-base-v2")
    config = AlbertConfig.from_pretrained(config_path, output_hidden_states=True)
    model = TFAlbertModel.from_pretrained(pytorch_dir, config=config, from_pt=True)
    e = albert_tokenizer.encode(text, max_length=512)
    model_input = tf.constant(e)[None, :]  # Batch size 1
    output = model(model_input)

    v = [0] * 768
    # generate sentence vectors by averaging the word vectors
    for i in range(1, len(model_input[0]) - 1):
        v = v + output[0][0][i].numpy()

    vector = v/len(model_input[0])
    return vector

However while loading the model, a warning comes up:

Some weights or buffers of the PyTorch model TFAlbertModel were not initialized from the TF 2.0 model and are newly initialized: ['predictions.LayerNorm.bias', 'predictions.dense.weight', 'predictions.LayerNorm.weight', 'sop_classifier.classifier.bias', 'predictions.dense.bias', 'sop_classifier.classifier.weight', 'predictions.decoder.bias', 'predictions.bias', 'predictions.decoder.weight'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

Can anyone tell me if I am doing anything wrong? What does the warning mean? I saw issue #5588 on the github repo of Transformers. Don't know if my issue is the same as this.

Upvotes: 0

Views: 2160

Answers (1)

longdragon
longdragon

Reputation: 51

I think you could try using

model = AlbertModel.from_pretrained

instead of

model = TFAlbertModel.from_pretrained

in the VectorizeSentence definition.

AlbertModel is the name of the class for the pytorch format model, and TFAlbertModel is the name of the class for the tensorflow format model.

I'm not sure exactly what load_tf_weights_in_albert() does, but I think that once you have done that your model is in pytorch format.

Upvotes: 1

Related Questions