CODEWITHSUNDEEP
pythonscikit-learnnlpldagensim
MachoMan
MachoMan

Reputation: 63

Perplexity comparision issue in SKlearn LDA vs Gensim LDA

I applied lda with both sklearn and with gensim. Then i checked perplexity of the held-out data.

I am getting negetive values for perplexity of gensim and positive values of perpleixy for sklearn. How do i compare those values.

sklearn perplexity = 417185.466838

gensim perplexity = -9212485.38144

Upvotes: 2

Views: 3986

Answers (1)

Lev Konst
Lev Konst

Reputation: 341

In order to compare perplexities you need to convert gensim's perplexity using np.exp(-1. * gensim_model.log_perplexity(train_corpus)).

See here for a general comparison of Gensim and Sklearn LDA implementations.

Upvotes: 2

Related Questions