CODEWITHSUNDEEP
rnlpquanteda
Kasi
Kasi

Reputation: 245

In R, combing individual word count and dictionary word count

I need to count words in a document. In some cases, I need to count specific words (e.g. "fresh"), in other cases I need to get the total count of a set of words ("philadelphia","aunt").

I know how do this in two separate steps (see code below), but how can I do this at the same time?

The code below counts specific words.

library("quanteda")
txt <- "In west Philadelphia born and raised On the playground was where I spent most of my days Chillin' out maxin' relaxin' all cool And all shootin some b-ball outside of the school When a couple of guys who were up to no good Started making trouble in my neighborhood I got in one little fight and my mom got scared."
tokens(txt) %>% tokens_select(c("trouble", "fight")) %>% dfm()

Output is:

trouble, fight
1, 1

The code below counts dictionary words and writes the total count to one column.

mydict <- dictionary(list(all_terms = c("chillin", "relaxin", "shootin")))
count <-dfm(txt,dictionary = mydict)

Output is:

all_terms
3

How can I combine the two?

I would like something like this: (code is hypothetical and does NOT work)

tokens(txt) %>% tokens_select(c("trouble", "fight"), mydict) %>% dfm()

or

tokens(txt) %>% tokens_select(c("trouble", "fight"), all_terms=c("chillin","relaxin","shootin")) %>% dfm()

Desired output:

trouble, fight, all_terms
1, 1, 3

Upvotes: 0

Views: 217

Answers (3)

Kohei Watanabe
Kohei Watanabe

Reputation: 880

This is what I suggested in the comment.

> library("quanteda")
> txt <- "In west Philadelphia born and raised On the playground was where I spent most of my days Chillin' out maxin' relaxin' all cool And all shootin some b-ball outside of the school When a couple of guys who were up to no good Started making trouble in     my neighborhood I got in one little fight and my mom got scared."
> dict <- dictionary(list(all_terms = c("chillin", "relaxin", "shootin")))
> dfmt <- dfm(txt)
> dfmt_dict <- dfm_lookup(dfmt, dict, exclusive = FALSE, cap = FALSE)
> topfeatures(dfmt_dict)
       in       and        of        my all_terms         '       the         i 
        3         3         3         3         3         3         2         2 
      all       got 
        2         2

Upvotes: 0

Ken Benoit
Ken Benoit

Reputation: 14902

There are a couple of ways, this is probably the simplest. Define a dictionary where the key is equal to the word value for each specific word, and a group key for sets of words -- in your example, "all_terms".

library("quanteda")
## Package version: 2.1.2

txt <- "In west Philadelphia born and raised On the playground was where I spent most of my days Chillin' out maxin' relaxin' all cool And all shootin some b-ball outside of the school When a couple of guys who were up to no good Started making trouble in my neighborhood I got in one little fight and my mom got scared."

dict <- dictionary(list(
  trouble = "trouble",
  fight = "fight",
  all_terms = c("chillin", "relaxin", "shootin")
))

Now when you compile the dfm, you will get what you are after.

dfmat <- dfm(txt, dictionary = dict)
dfmat
## Document-feature matrix of: 1 document, 3 features (0.0% sparse).
##        features
## docs    trouble fight all_terms
##   text1       1     1         3

To coerce this to a simpler object, including the output you listed, you can do this:

# as a named numeric vector
structure(as.vector(dfmat), names = featnames(dfmat))
##   trouble     fight all_terms 
##         1         1         3

# per your output
cat(
  paste(featnames(dfmat), collapse = ", "), "\n",
  paste(as.vector(dfmat), collapse = ", ")
)
## trouble, fight, all_terms 
##  1, 1, 3

Note that it's not a good idea (as in the other answer) to access the object internals directly. Use extractor functions such as featnames() instead.

Added:

An alternative way without creating the named list of items:

dict <- dictionary(list(all_terms = c("chillin", "relaxin", "shootin")))
single_words <- c("trouble", "fight")

tokens(txt) %>%
  tokens_lookup(dictionary = dict, exclusive = FALSE) %>%
  tokens_keep(pattern = c(names(dict), single_words)) %>%
  dfm()
## Document-feature matrix of: 1 document, 3 features (0.0% sparse).
##        features
## docs    all_terms trouble fight
##   text1         3       1     1

Upvotes: 1

Tech Commodities
Tech Commodities

Reputation: 1959

Is brevity important, i.e. having it all on one line? If not, a solution is to extract the data from the dfm objects and then combine into the form you're after - matrix, data.frame, tibble.

library("quanteda")
library(magritte) # for the pipe
txt <- "In west Philadelphia born and raised On the playground was where I spent most of my days Chillin' out maxin' relaxin' all cool And all shootin some b-ball outside of the school When a couple of guys who were up to no good Started making trouble in     my neighborhood I got in one little fight and my mom got scared."
mydict <- dictionary(list(all_terms = c("chillin", "relaxin", "shootin")))

first <-  dfm(tokens_select(tokens(txt), c("trouble", "fight")))
second <- dfm(txt,dictionary = mydict)

# These are the outputs you're after
first@Dimnames$features
first@x

second@Dimnames$features
second@x

# Combine into a matrix
 matrix(c(first@Dimnames$features, second@Dimnames$features), ncol = 3) %>% 
   rbind(c(first@x, second@x))

# Or make two vectors for use elsewhere
  paste(c(first@Dimnames$features, second@Dimnames$features), collapse = ", ")
  paste(c(first@x, second@x), collapse = ", ")

Upvotes: 1

Related Questions