CODEWITHSUNDEEP
listtmcorpus
Benjamin Levy
Benjamin Levy

Reputation: 343

Removing words from a corpus of documents with a tailored list of words

The tm package has the ability to let the user 'prune' the words and punctuation in a corpus of documents: tm_map( corpusDocs, removeWords, stopwords("english") )

Is there a way to supply tm_map with a tailored list of words that is read in from a csv file and used in place of stopwords("english")?

Thank you.

BSL

Upvotes: 0

Views: 198

Answers (2)

Vezir
Vezir

Reputation: 101

Lets take a file (wordMappings)

"from"|"to"
###Words######
"this"|"ThIs"
"is"|"Is"
"a"|"A"
"sample"|"SamPle"

First removel of words;

readFile <- function(fileName, seperator) {
  read.csv(paste0("data\\", fileName, ".txt"), 
                             sep=seperator, #"\t", 
                             quote = "\"",
                             comment.char = "#",
                             blank.lines.skip = TRUE,
                             stringsAsFactors = FALSE,
                             encoding = "UTF-8")

}

kelimeler <- c("this is a sample")
corpus = Corpus(VectorSource(kelimeler))
seperatorOfTokens <- ' '
words <- readFile("wordMappings", "|")

toSpace <- content_transformer(function(x, from) gsub(sprintf("(^|%s)%s(%s%s)", seperatorOfTokens, from,'$|', seperatorOfTokens, ')'), sprintf(" %s%s", ' ', seperatorOfTokens), x))
for (word in words$from) {
  corpus <- tm_map(corpus, toSpace, word)
}

If you want a more flexible solution, for example not just removing also replacing with then;

#Specific Transformations
toMyToken <- content_transformer( function(x, from, to)
  gsub(sprintf("(^|%s)%s(%s%s)", seperatorOfTokens, from,'$|', seperatorOfTokens, ')'), sprintf(" %s%s", to, seperatorOfTokens), x))

for (i in seq(1:nrow(words))) {
  print(sprintf("%s -> %s ", words$from[i], words$to[i]))
  corpus <- tm_map(corpus, toMyToken, words$from[i], words$to[i])
}

Now a sample run;

[1] "this -> ThIs "
[1] "is -> Is "
[1] "a -> A "
[1] "sample -> SamPle "
> content(corpus[[1]])
[1] " ThIs Is A SamPle "
>

Upvotes: 1

Benjamin Levy
Benjamin Levy

Reputation: 343

My solution, which may be cumbersome and inelegant:

#read in items to be removed
removalList = as.matrix( read.csv( listOfWordsAndPunc, header = FALSE ) )
#
#create document term matrix
termListing = colnames( corpusFileDocs_dtm )
#
#find intersection of terms in removalList and termListing
commonWords = intersect( removalList, termListing )
removalIndxs = match( commonWords, termListing )
#
#create m for term frequency, etc.
m = as.matrix( atsapFileDocs_dtm )
#
#use removalIndxs to drop irrelevant columns from m
allColIndxs = 1 : length( termListing )
keepColIndxs = setdiff( allColIndxs, removalIndxs )
m = m[ ,keepColIndxs ]
#
#thence to tf-idf analysis with revised m

Any stylistic or computational suggestions for improvement are gratefully sought.

BSL

Upvotes: 0

Related Questions