Arun
Arun

Reputation: 625

How to remove punctuation using tm package without removing a period in R?

I am using tm package to remove the punctuation. When ever I dont have a space between the period and the following word, the remove punctuation just removes the period and concatenates the previous word

For example:

"transactions.Proceessed"  
"trnsaction.It"

After applying "remove punctuation" using tm package I am getting the output as below:

"transactionsProceessed"  
"trnsactionIt"

Is it possible to have spaces between the words or to retain the period inspite of using remove punctuation function?

Update

The example provided is a sample one . The input file is a huge text file. I am using tm_map function to remove the punctuations. This is the code that i use

# set parameters
candidates <- c("Obama", "Romney")
pathname <- "H:/datasets/"

# clean texts
cleanCorpus <- function(corpus){
  #corpus.tmp <- tm_map(corpus, removePunctuation)
  ##corpus.tmp <- gsub(".", " ", corpus, fixed = TRUE)
  f <- content_transformer(function(x, pattern) sub(pattern, " ", s.cor  ))
  corpus.tmp <- tm_map(s.cor, f, "[[:punct:]]")
  corpus.tmp <- tm_map(corpus.tmp, stripWhitespace)
  corpus.tmp <- tm_map(corpus.tmp, content_transformer(tolower))
  ##corpus.tmp <- tm_map(corpus.tmp, stemDocument)
  corpus.tmp <- tm_map(corpus.tmp, removeWords, stopwords("english"))
  return(corpus.tmp)
}
# create text document matrix
generateTDM <- function(cand, path){
  s.dir <- sprintf("%s/%s", path, cand)
  s.cor <- Corpus(DirSource(directory = s.dir, encoding = "UTF-8"))
  s.cor.cl <- cleanCorpus(s.cor)
  s.tdm <- TermDocumentMatrix(s.cor.cl)
  s.tdm <- removeSparseTerms(s.tdm, 0.7)
  result <- list(name = cand, tdm = s.tdm)
}
# execute function and create a Text Document Matrix
tdm <- lapply(candidates, generateTDM, path = pathname)

............................................................................

Upvotes: 2

Views: 3229

Answers (1)

erasmortg
erasmortg

Reputation: 3278

This (these) solution(s) apply to your first option (removing the full stop, and in general, all punctuation and adding a space in between):

If your input is as simple as your example, you could try sub from base:

sub(".", " ", "transactions.Proceessed", fixed=TRUE)
#[1] "transactions Proceessed"
sub(".", " ", "trnsaction.It", fixed=TRUE)
#[1] "trnsaction It"
x <- c("transactions.Processed", "trnsaction.It")
sub(".", " ", x, fixed=TRUE)
#[1] "transactions Processed" "trnsaction It"
#this one should remove all punctuation
sub("[[:punct:]]", " ",x)
#[1] "transactions Processed" "trnsaction It" 

The idea for object of class VCorpus or Corpus is the same, but you would have to use content_transformer to do so:

#You would have to switch to your actual corpus
x <- c("transactions.Processed", "trnsaction.It")
sub("[[:punct:]]", " ",x)
#[1] "transactions Processed" "trnsaction It"         
xx <- VCorpus(VectorSource(x))
f <- content_transformer(function(x, pattern) sub(pattern, " ", x))
xx2 <- tm_map(xx, f, "[[:punct:]]")

Here you can see the full structure:

xx2
#    List of 2
# $ 1:List of 2
#  ..$ content: chr "transactions Processed"
#  ..$ meta   :List of 7
#  .. ..$ author       : chr(0) 
#  .. ..$ datetimestamp: POSIXlt[1:1], format: "2015-09-29 09:24:42"
#  .. ..$ description  : chr(0) 
#  .. ..$ heading      : chr(0) 
#  .. ..$ id           : chr "1"
#  .. ..$ language     : chr "en"
#  .. ..$ origin       : chr(0) 
#  .. ..- attr(*, "class")= chr "TextDocumentMeta"
#  ..- attr(*, "class")= chr [1:2] "PlainTextDocument" "TextDocument"
# $ 2:List of 2
#  ..$ content: chr "trnsaction It"
#  ..$ meta   :List of 7
#  .. ..$ author       : chr(0) 
#  .. ..$ datetimestamp: POSIXlt[1:1], format: "2015-09-29 09:24:42"
#  .. ..$ description  : chr(0) 
#  .. ..$ heading      : chr(0) 
#  .. ..$ id           : chr "2"
#  .. ..$ language     : chr "en"
#  .. ..$ origin       : chr(0) 
#  .. ..- attr(*, "class")= chr "TextDocumentMeta"
#  ..- attr(*, "class")= chr [1:2] "PlainTextDocument" "TextDocument"
# - attr(*, "class")= chr [1:2] "VCorpus" "Corpus"

Or just the content:

xx2[[1]][1]
#$content
#[1] "transactions Processed"

xx2[[2]][1]
#$content
#[1] "trnsaction It"

Upvotes: 3

Related Questions