CODEWITHSUNDEEP
rweb-scrapingrvest
Brad
Brad

Reputation: 9

Scraping with Rvest in R Studio: Returns df 0 rows by 32 columns

I am trying to scrape some sports data from this website (https://en.khl.ru/stat/players/1097/skaters/) using rvest. There are no pages to filter through, but there is a 'Show All' icon to show all the data on the page.

I have been trying to use a css selector to extract the table. Unfortunately, no rows are produced but the column names of the table are present.

I suspect the problem lies in the website's interactive features with the table.

Upvotes: 0

Views: 85

Answers (1)

Dave2e
Dave2e

Reputation: 24079

Yes, this page is dynamically generated, thus troublesome for rvest to handle. But the key to scrape this page is to realize the data is stored as JSON in a script element on the page.

The code below reads the page and extracts the script nodes. Reviewed the script node to find the correct one. Then some trial and error extracted the JSON data. Cleaned up the player and team name columns for the final answer.

library(rvest)
library(dplyr)
library(stringr)

url <- "https://en.khl.ru/stat/players/1097/skaters/" 
page <- read_html(url)

#the data for the page is stored in a script element
scripts <-page %>% html_elements("script")
#get column names
headers <- page %>% html_elements("thead th") %>% html_text()

#examined the nodes and manually determined the 31st node was it 
tail(scripts, 18)
data <- scripts[31] %>% html_text()

#examined the data string and notice the start of the JSON was '[ ['
#end of the JSON was ']]'
jsonstring <- str_extract(data, "\\[ \\[.+\\]\\]")

#convert the JSON into data frame
answer <- jsonlite::fromJSON(jsonstring) %>% as.data.frame

#rename column titles
names(answer) <- headers

#function to clean up html code in columns
cleanhtml <- function(text) {
   out<-text %>% read_html() %>% html_text()
}

#remove the html information in columns 1 &3
answer <- answer[ , -32] %>% rowwise() %>% 
             mutate(Player = cleanhtml(Player), Team=cleanhtml(Team))
answer

Upvotes: 1

Related Questions