Extract table from a JavaScript based webpage

Question

I want to extract the table under the Option Chain tab of the following webpage using R -

https://nseindia.com/get-quotes/derivatives?symbol=SBIN

I am trying to use rvest for this purpose, however the standard way of extracting tables from simple html based webpages don't seem to be working.

The following is what I'm trying -

library('rvest')
x <- c('https://nseindia.com/get-quotes/derivatives?symbol=SBIN')
url <- paste(x,collapse="")
webpage <- read_html(url)
data <- webpage %>% html_nodes(xpath = '//*[@id="optionChainTable-equity"]') %>% html_table()

This is returning an empty list. What am I missing?

Answer 1

The data you are looking for is loaded dynamically as JSON from a different url. Also, I seem to need to change user agent to get at the data otherwise the server refuses the connection.

Once you have it, you need to parse the JSON with a library like jsonlite. In this example I have converted the data to a tibble to make it easier to see in the console.

library(httr)
library(jsonlite)
library(tibble)

user_agent    <-  paste("Mozilla/5.0 (Windows NT 6.1;",
                         "rv =61.0) Gecko/20100101 Firefox/61.0");

accept_string <-  paste0("text/html,application/xhtml+xml,",
                          "application/xml;q=0.9,*/*;q=0.8");

headers       <- add_headers( `User-Agent`                = user_agent,
                              Accept                      = accept_string,
                              `Accept-Language`           = "en-GB,en;q=0.5",
                              `Accept-Encoding`           = "gzip, deflate",
                              Connection                  = "keep-alive",
                              `Upgrade-Insecure-Requests` = "1");

url <- "https://nseindia.com/api/option-chain-equities?symbol=SBIN"
table_contents <- content(GET(url, config = headers), "text")
df <- as_tibble(fromJSON(table_contents)$`records`$data)

df
# A tibble: 89 x 4
#>    strikePrice expiryDate PE$strikePrice $expiryDate $underlying $identifier
#>          <int> <chr>               <int> <chr>       <chr>       <chr>
#>  1         210 30-Jan-20~            210 30-Jan-2020 SBIN        OPTSTKSBIN~
#>  2         215 30-Jan-20~            215 30-Jan-2020 SBIN        OPTSTKSBIN~
#>  3         220 30-Jan-20~            220 30-Jan-2020 SBIN        OPTSTKSBIN~
#>  4         225 30-Jan-20~            225 30-Jan-2020 SBIN        OPTSTKSBIN~
#>  5         230 30-Jan-20~            230 30-Jan-2020 SBIN        OPTSTKSBIN~
#>  6         235 30-Jan-20~            235 30-Jan-2020 SBIN        OPTSTKSBIN~
#>  7         240 30-Jan-20~            240 30-Jan-2020 SBIN        OPTSTKSBIN~
#>  8         245 30-Jan-20~            245 30-Jan-2020 SBIN        OPTSTKSBIN~
#>  9         250 27-Feb-20~            250 27-Feb-2020 SBIN        OPTSTKSBIN~
#> 10         250 26-Mar-20~            250 26-Mar-2020 SBIN        OPTSTKSBIN~
# ... with 79 more rows, and 34 more variables: $openInterest <int>,
#   $changeinOpenInterest <int>, $pchangeinOpenInterest <dbl>,
#   $totalTradedVolume <int>, $impliedVolatility <dbl>, $lastPrice <dbl>,
#   $change <dbl>, $pChange <dbl>, $totalBuyQuantity <int>,
#   $totalSellQuantity <int>, $bidQty <int>, $bidprice <dbl>, $askQty <int>,
#   $askPrice <dbl>, $underlyingValue <int>, CE$strikePrice <int>,
#   $expiryDate <chr>, $underlying <chr>, $identifier <chr>, $openInterest <int>,
#   $changeinOpenInterest <int>, $pchangeinOpenInterest <dbl>,
#   $totalTradedVolume <int>, $impliedVolatility <dbl>, $lastPrice <dbl>,
#   $change <dbl>, $pChange <dbl>, $totalBuyQuantity <int>,
#   $totalSellQuantity <int>, $bidQty <int>, $bidprice <dbl>, $askQty <int>,
#   $askPrice <dbl>, $underlyingValue <int>