CODEWITHSUNDEEP
deep-learninggensimword2vecloss-function
jpmd
jpmd

Reputation: 11

Gensim word2vec model loss becomes 0 after few epochs

Trying to build gensim word2vec model. Corpus contains 1 M sentences. I am using callback to print loss after every epochs. After few epochs loss becomes zero. Any idea why loss becomes 0?

Loss after epoch 0: 17300302.0
Loss after epoch 1: 11381698.0
Loss after epoch 2: 8893964.0
Loss after epoch 3: 7105532.0
           ...
           ...
           ...
Loss after epoch 54: 1283432.0
Loss after epoch 55: 1278048.0
Loss after epoch 56: 316968.0
Loss after epoch 57: 0.0
Loss after epoch 58: 0.0
Loss after epoch 59: 0.0
Loss after epoch 60: 0.0
Loss after epoch 61: 0.0
Loss after epoch 62: 0.0
Loss after epoch 63: 0.0
Loss after epoch 64: 0.0
Loss after epoch 65: 0.0
Loss after epoch 66: 0.0

Upvotes: 1

Views: 1151

Answers (1)

gojomo
gojomo

Reputation: 54153

Update: There's a problem with the precision of the type being used for this (also suboptimal) multi-epoch running-tally. See gensim bug #2735 for discussion.

--

Previous answer:

It's hard to say without seeing the code, and having some ideas about the data, you're using to do training.

However note:

  • the loss as reported by gensim Word2Vec currently isn't a per-epoch loss, but a running total since the start of the last call to .train(). (See gensim project bug #2617, and the mentioned pull-requests/issues there.) To get the per-epoch loss – a value you'd expect to go down for a while, then stagnate – you'd have to undo this tally by comparing the reported value to the previously-reported value. (Are you doing that?)

  • the only two ways I can think of the per-epoch loss actually going to zero would be (1) some sort of error in the corpus iterator, so no new training is actually happening per epoch – this might be evident in logging output, if you enable INFO level logging; (2) extreme overfitting – a model way too large for the variety in the actual training data – in which case maybe the model could truly become "perfect" at its nearby-word prediction task.

There was one previous report of similar behavior on the gensim project support list – see https://groups.google.com/d/msg/gensim/IH5-nWoR_ZI/Hg0qvLYiAAAJ - but after I'd made some suggestions & asked some followup questions, the original reporter never followed up. So it's unclear what error they'd made in their code, or whether there might be a bug in gensim. Still, you may find the questions & comments in that thread useful.

If you can't otherwise solve your problem, it would help to see in your question more details of how you're training the model (code, parameters, & data), & how you're reporting-out those loss numbers, together with any other details as asked about in that thread.

Upvotes: 1

Related Questions