CODEWITHSUNDEEP
pythonpytorch
Beverlie
Beverlie

Reputation: 461

How to Train Word Embedding With Pytorch

I am trying learn and practice how to train the embedding of Vocabulary set using pytorch.

https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html

loss = loss_function(log_probs, torch.tensor([word_to_ix[target]], dtype=torch.long))

At the above tutorial example, the loss was calculated between log_probs which is 4 x 10 tensor (4 for context words number and 10 is embedding_dimension) and the word index of target which is integer ranged from 0 to 49.

I can't understand why the code doesn't compare between its context embedding with its target embedding but just compare with its class index, which is a mere integer holding no information..

I think one must go back to the embedding parameter then call it, and compare it with its context I guess.

Is it just because it's only for the tutorial or I am misinterpreting some point?

Thanks for your help in advance.

Upvotes: 0

Views: 2427

Answers (1)

lintang
lintang

Reputation: 100

The embedding is a by-product of training your model. The model itself is trained with supervised learning to predict the next word give the context words. This is usually done (also in that tutorial) in the form of a one-hot encoder. The log_probs are the output of the model that are probabilities in logarithmic form which is then compared to one-hot encoder target. Higher probabilities for the correct target word corresponds to lower loss and inversely, lower probabilities for correct target words will cause larger loss signals to propagate through the network and change the weights. That is why the log_probs are compared against the target.

Upvotes: 1

Related Questions