CODEWITHSUNDEEP
pythonpytorchtorch
Yingqiang Gao
Yingqiang Gao

Reputation: 999

Pytorch: use pretrained vectors to initialize nn.Embedding, but this embedding layer is not updated during the training

I initialized nn.Embedding with some pretrain parameters (they are 128 dim vectors), the following code demonstrates how I do this:

self.myvectors = gensim.models.KeyedVectors.load_word2vec_format(cfg.vec_dir)
self.vec_weights = torch.FloatTensor(self.myvectors.vectors)
self.embeds = torch.nn.Embedding.from_pretrained(self.vec_weights)

cfg.vec_dir is a json file where vec_dir indicates the path of the pretrained 128 dim vectors I used to initialize this layer.

After the model is trained, I print out this embedding layer, and I found that the parameters are exactly the same as I initialized them, so clearly the parameters are not updated during the training. Why is this happening? What should I do in order to update these vectors?

Upvotes: 2

Views: 4773

Answers (1)

Justin O Barber
Justin O Barber

Reputation: 11591

The torch.nn.Embedding.from_pretrained classmethod by default freezes the parameters. If you want to train the parameters, you need to set the freeze keyword argument to False. See the documentation.

So you might try this instead:

self.embeds = torch.nn.Embedding.from_pretrained(self.vec_weights, freeze=False)

Upvotes: 4

Related Questions