CODEWITHSUNDEEP
rrcurl
Charlie
Charlie

Reputation: 11

Scraping does not return the desired data

I am trying to get data from the site https://bill.torrentpower.com/. I desire to input the city "Ahmedabad" and Service number "3031629" and extract the table which gives the bill details.

My code is simple

a<- postForm("https://bill.torrentpower.com/billdetails.aspx",
    "ctl00$cph1$drpCity" = 1,
    "ctl00$cph1$txtServiceNo" =  "3031629",
    .opts = list(ssl.verifypeer = FALSE)
)
write(a,file="a.html")

When I open the file a.html, I do not see the table containing the bill details. All other details are visible on a.html. My aim is to capture the tablular output as an R object.

Upvotes: 1

Views: 62

Answers (1)

biobirdman
biobirdman

Reputation: 4120

The issue here is that the table is generated by the JavaScript code after the page has loaded and hence you will not get the content of the table.

This is a common problem with scraping information that has lots of dynamic content.

A work around this is to stimulate a web browser using RSelenium. http://cran.r-project.org/web/packages/RSelenium/RSelenium.pdf

This will stimulate with web browser in your R session and you can navigate the webpages using various methdos ( see the user manual for info)

Personally, I find RSelenium with PhantomJS combination the most useful since I use a lot of JavaScript. Alternatively, if you find using R Syntax abit troublesome you may use PhantomJS on its own as well. http://phantomjs.org/

Upvotes: 1

Related Questions