CODEWITHSUNDEEP
rregextagscorpusquanteda
Wangana
Wangana

Reputation: 233

R: quanteda removing tags from corpus

I am working with a number texts using the quanteda package. My texts contain tags in them, some with unique values like URLs. I want remove not only the tags but everything inside the tags as well.

Example:

<oa>
</oa>
<URL: http://in.answers.yahoo.com/question/index;_ylt=Ap2wvXm2aeRQKHO.HeDgTfneQHRG;_ylv=3?qid=1006042400700>
<q>
<ad>
</ad>

I'm not sure how to remove them while working with the quanteda package. It seems to me like the dfm function would be the place to use it, I don't think stopwords will work because of the unique URLs. I can use the following gsub with regex expression to successfully target the tags I want to remove: 

x <- gsub("<.*?>", "", y)

I've gone through the gfm documentation and have tried a few things with the remove and value type arguments, but perhaps I don't understand the documentation very well.

Also as shown by the answer in this question I tried the dfm_select function but no dice as well.

Here is my code:

library(readtext)
library(quanteda)

#create directory
data_dir <- list.files(pattern="*.txt", recursive = TRUE, full.names = TRUE)

#create corpus    
micusp_corpus <- corpus(readtext(data_dir))

#add field 'region'
docvars(micusp_corpus, "Region") <- gsub("(\\w{6})\\..*?$", "", rownames(micusp_corpus$documents))

#create document feature matrix
micusp_dfm <- dfm(micusp_corpus, groups = "Region", remove_punct = TRUE, remove_numbers = TRUE, remove_symbols = TRUE)
 #try to remove tags       
micusp_dfm <- dfm_select(micusp_dfm, "<.*?>", selection = "remove", valuetype = "regex")

#show top tokens (note the appearence of the tag content "oa")
textstat_frequency(micusp_dfm, n=10)

Upvotes: 1

Views: 805

Answers (1)

Ken Benoit
Ken Benoit

Reputation: 14902

While your question does not provide a reproducible example, I think I can help. You want to clean the texts that go into your corpus, before you reach the dfm construction stage. Replace the #create corpus line with this:

# read texts, remove tags, and create the corpus
tmp <- readtext(data_dir)
tmp$text <- gsub("<.*?>", "", tmp$text)
micusp_corpus <- corpus(tmp)

Upvotes: 2

Related Questions