CODEWITHSUNDEEP
rtext
Drew
Drew

Reputation: 135

Efficient way to remove all proper names from corpus

Working in R, I'm trying to find an efficient way to search through a file of texts and remove or replace all instances of proper names (e.g., Thomas). I assume there is something available to do this but have been unable to locate.

So, in this example the words "Susan" and "Bob" would be removed. This is a simplified example, when in reality would want this to apply to hundreds of documents and therefore a fairly large list of names. 

texts <- as.data.frame (rbind (
    'This text stuff if quite interesting',
    'Where are all the names said Susan',
   'Bob wondered what happened to all the proper nouns'
    ))
names(texts) [1] <- "text"

Upvotes: 3

Views: 3873

Answers (1)

lukeA
lukeA

Reputation: 54237

Here's one approach based upon a data set of firstnames: 

install.packages("gender") 
library(gender)
install_genderdata_package()

sets <- data(package = "genderdata")$results[,"Item"]
data(list = sets, package = "genderdata")
stopwords <- unique(kantrowitz$name)

texts <- as.data.frame (rbind (
  'This text stuff if quite interesting',
  'Where are all the names said Susan',
  'Bob wondered what happened to all the proper nouns'
))

removeWords <- function(txt, words, n = 30000L) {
  l <- cumsum(nchar(words)+c(0, rep(1, length(words)-1)))
  groups <- cut(l, breaks = seq(1,ceiling(tail(l, 1)/n)*n+1, by = n))
  regexes <- sapply(split(words, groups), function(words) sprintf("(*UCP)\\b(%s)\\b", paste(sort(words, decreasing = TRUE), collapse = "|")))
  for (regex in regexes)  txt <- gsub(regex, "", txt, perl = TRUE, ignore.case = TRUE)
  return(txt)
}
removeWords(texts[,1], stopwords)
# [1] "This text stuff if quite interesting"           
# [2] "Where are all the names said "                  
# [3] " wondered what happened to all the proper nouns"

It may need some tuning for your specific data set.

Another approach could be based upon part-of-speech tagging.

Upvotes: 2

Related Questions