CODEWITHSUNDEEP
javascriptrweb-scraping
Maciej
Maciej

Reputation: 3303

Scraping javascript with R

I want to download tables from metal-archives.com, exactly from http://www.metal-archives.com/artist/rip, but there is one big problem. This tables are generated by javascript. In fact I don't know what to do in this case.

Is there a possibility to parse this site with R and XML package?

Upvotes: 3

Views: 2087

Answers (2)

Maciej
Maciej

Reputation: 3303

Thanks to user bubmu I achieve what I wanted. Below is code, which solves my problem.

a<-1:8
b<-200*a
x<-paste("http://www.metal-archives.com/artist/ajax-rip?iDisplayStart=",b,"&sEcho=",a,sep="")
x<-c(x,"http://www.metal-archives.com/artist/ajax-rip?iDisplayStart=1700&sEcho=9")

JSONparse<-function(x){
  library(XML)
  doc<-htmlParse(x)
  str<-xpathApply(doc,'//p',xmlValue)[[1]][1]
  x1<-strsplit(str,'\\[')
  x1<-x1[[1]][-1]
  x1<-x1[-1]

  x2<-strsplit(x1,'\\",')
  x3<-lapply(x2, function(y) {
    y<-gsub('\\t','',y)
    y<-gsub('\\n','',y)
    y<-gsub('\\r','',y)
    y<-gsub('\\\"','',y)
    y<-gsub('\\]}','',y)
    y<-gsub('\\],','',y)
    y<-as.data.frame(t(y))
    y})

  allinall<-do.call('rbind',x3)
  colnames(allinall)<-c("Artist","Country","Band","When","Why")
  allinall
}

metallum<-lapply(x,JSONparse)
metallum<-do.call('rbind',metallum)

But it works only for this site. Of course better is RJSONIO or rjson package.

Upvotes: 2

bumbu
bumbu

Reputation: 1317

Here's all information in JSON format

http://www.metal-archives.com/artist/ajax-rip

Upvotes: 7

Related Questions