CODEWITHSUNDEEP
rweb-scrapingrvest
Avijit Mallick
Avijit Mallick

Reputation: 37

Rvest Web scraping after selecting multiple historical dates in R

I am trying to download the data from the HTML website https://wrldc.in/GeneratorScheduleChart.html#

I have been using the R package Rvest

I have seen some of the examples in StackOverflow from the link request_POST.

While running this code I am getting error that Error: object 'request_POST' not found

I am using selector gadget tools to get the HTML and CSS selector.

here we have another dropdown menu under 'Last Update at' header Last Update at that contains the data list @ 1-minute intervals.

What I want is to download the data after selecting calendar dates Calendar dates/ datepicker or date pickers and then get all the tables of all the list of 'last Updates at'

I do not like to use Rselenium package for this.

Upvotes: 0

Views: 226

Answers (1)

QHarr
QHarr

Reputation: 84455

You can see the correct endpoint dictated in the Javascript for the Ajax calls in the source html. If you then find the call to that endpoint, in the network tab, you can recreate the request with httr. You only need the Content-Type header but I would also add the User-Agent. In the body you pass the data.

library(httr)
library(jsonlite)

headers = c('User-Agent' = 'Mozilla/5.0', 'Content-Type' = 'application/json; charset=UTF-8')
data = '{date:"2021-7-19"}'

r <- httr::POST(url = 'https://wrldc.in/GeneratorSchedule_data.aspx/Get_GeneratorScheduleData_state_Wise',
                httr::add_headers(.headers=headers), body = data) %>% content()

df <- jsonlite::parse_json(r$d, simplifyVector = T)

Upvotes: 1

Related Questions