CODEWITHSUNDEEP
pythonpytorch
Shengtian
Shengtian

Reputation: 31

Should the embedding layer be changed during training a neural network?

I'm a new one for the field of deep learning and Pytorch.

Recently when I learn one of the pytorch tutorial example for NER task, I found the embedding of nn.Embedding changed during the training.

So my question is should the embedding be changed during training the network?

And if I want to load a pre-trained embedding (for example, trained word2vec embedding) into a PyTorch embedding layer, should the pre-trained embedding also be changed during the train process?

Or how could I prevent updating the embeddings?

Thank you.

Upvotes: 3

Views: 1361

Answers (1)

Szymon Maszke
Szymon Maszke

Reputation: 24726

One can either learn embeddings during the task, finetune them for task at hand or leave as they are (provided they have been learned in some fashion before).

In the last case, with standard embeddings like word2vec one eventually finetunes (using small learning rate), but uses vocabulary and embeddings provided. When it comes to current SOTA like BERT fine-tuning on your data should always be done, but in unsupervised way (as trained originally).

Easiest way to use them is static method of torch.nn.Embedding.from_pretrained (docs) and provide Tensor with your pretrained data.

If you want the layer to be trainable, pass freeze=False, by default it's not as you want.

Upvotes: 2

Related Questions