CODEWITHSUNDEEP
rquantedatopicmodels
fritsvegters
fritsvegters

Reputation: 57

quanteda convert to topicmodels retaining docvars

I'm using the awesome quanteda package to convert my dfm to a topicmodels format. However, in the process I'm losing my docvars which I need for identifying which topics are most likely prevalent in my documents. This is especially a problem given that topicmodels package (as does STM) only selects non-zero counts. The number of documents in the original dfm and the model output hence differ. Is there any way for me to correctly identify the documents in casu?

Upvotes: 0

Views: 753

Answers (3)

Kohei Watanabe
Kohei Watanabe

Reputation: 880

I don't think the problem is described clearly, but I believe I understand what it is.

Topic models' document feature matrix cannot contain empty documents, so they return named vector of topics without these. But you can still live with it if you match them to the document names:

# mx is a quanteda's dfm
# topic is a named vector for topics from LDA

docvars(mx, "topic") <- topic[match(docnames(mx), names(topic))]

Upvotes: 1

phiver
phiver

Reputation: 23608

I checked your outcome. Because of your select statement you have no features left in dfm_speeches. Convert that to the "dtm" format as used by the topicmodels and you indeed get a document term matrix that has no documents and no terms.

But if your selection with dfm_select results in a dfm with features and you then convert it into a dtm format you will see docvars appearing.

dfm_speeches <- dfm(data_corpus_irishbudget2010, 
                    remove_punct = TRUE, remove_numbers = TRUE, remove = stopwords("english")) %>% 
  dfm_trim(min_termfreq = 4, max_docfreq = 10)

dfm_speeches <- dfm_select(dfm_speeches, c("Bruton", "Cowen"))

docvars(dfm_speeches)

dfmlda <- convert(dfm_speeches, to = "topicmodels")

This will then work further with topicmodels. I will admit that if you convert to a dtm for tm and you have no features you will see the documents appearing in the dtm. I'm not sure if there is a unintended side effect with the conversion to topicmodels if there are no features.

Upvotes: 1

fritsvegters
fritsvegters

Reputation: 57

Sorry, here's an example.

dfm_speeches <- dfm(data_corpus_irishbudget2010, 
            remove_punct = TRUE, remove_numbers = TRUE, remove = stopwords("english")) %>% 
dfm_trim(min_termfreq = 4, max_docfreq = 10)
dfm_speeches <- dfm_select(dfm_speeches, c("corbyn", "hillary"))

library(topicmodels)
dfmlda <- convert(dfm_speeches, to = "topicmodels") %>% 
dfmlda

As you can see, the dfmlda object is empty because the fact that I modified my dfm by removing specific words.

Upvotes: 0

Related Questions