CODEWITHSUNDEEP
documentgensimpredictlda
Feng Chen
Feng Chen

Reputation: 2253

How to use gensim topic modeling to predict new document?

I am new to gensim topic modeling. Here is my sample code:

import nltk
nltk.download('stopwords')
import re
from pprint import pprint
# Gensim
import gensim
import gensim.corpora as corpora
from gensim.utils import simple_preprocess
from gensim.models import CoherenceModel
# spacy for lemmatization
import spacy
# Plotting tools
import pyLDAvis
import pyLDAvis.gensim  # don't skip this
import matplotlib.pyplot as plt
#%matplotlib inline
# Enable logging for gensim - optional
import logging
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.ERROR)
import warnings
warnings.filterwarnings("ignore",category=DeprecationWarning)
# NLTK Stop words
from nltk.corpus import stopwords
stop_words = stopwords.words('english')
stop_words.extend(['from', 'subject', 're', 'edu', 'use'])
train=pd.DataFrame({'text':['find the most representative document for each topic',
                            'topic distribution across documents',
                            'to help with understanding the topic',
                            'one of the practical application of topic modeling is to determine']})
text=pd.DataFrame({'text':['how to find the optimal number of topics for topic modeling']})

data =  train.loc[:,'text'].values.tolist()

def sent_to_words(sentences):
    for sentence in sentences:
        yield(gensim.utils.simple_preprocess(str(sentence), deacc=True))

data_words = list(sent_to_words(data))
id2word = corpora.Dictionary(data_words)
corpus = [id2word.doc2bow(text) for text in data_words]
lda_model = gensim.models.ldamodel.LdaModel(corpus=corpus,
                                            id2word=id2word,
                                            num_topics=3)

So far so good. But I want to use lda_model to predict text. I at least need to know topic distribution over text and all the topic-word relation.

I think prediction is very common and important function for lda. But I do not know where I can find such function in gensim. Some answers says doc_lda = model[doc_bow] is prediction (Calculating topic distribution of an unseen document on GenSim). But I am not sure about it.

Upvotes: 0

Views: 7078

Answers (2)

Satyam Prasad Tiwari
Satyam Prasad Tiwari

Reputation: 1

This will help you:

new_doc=your processed text
new_doc_bow = dictionary.doc2bow(new_doc)
ldamodel.get_document_topics(new_doc_bow)

Upvotes: 0

qaiser
qaiser

Reputation: 2868

import pandas as pd 
train=pd.DataFrame({'text':['find the most representative document for each topic',
                        'topic distribution across documents',
                        'to help with understanding the topic',
                        'one of the practical application of topic modeling is to determine']})
text=pd.DataFrame({'text':['how to find the optimal number of topics for topic modeling']})


def sent_to_words(sentences):
for sentence in sentences:
    yield(gensim.utils.simple_preprocess(str(sentence), deacc=True))

#using your train data to train the model with 4 topics

data_words = list(sent_to_words(train['text']))
id2word = corpora.Dictionary(data_words)
corpus = [id2word.doc2bow(text) for text in data_words]

lda_model = gensim.models.ldamodel.LdaModel(corpus=corpus,
                                        id2word=id2word,
                                        num_topics=4)

#  predicting new text which is in text dataframe  
new_text_corpus =  id2word.doc2bow(text['text'][0].split())
lda[new_text_corpus]

#op

Out[75]:
[(0, 0.5517368), (1, 0.38150477), (2, 0.032756805), (3, 0.03400166)]

Upvotes: 2

Related Questions