CODEWITHSUNDEEP
rrvestrselenium
youraz
youraz

Reputation: 483

How to scrape infinite scroll page in R?

I'm trying to get some pieces of information on a website but it has an infinite scroll page. Is rvest package suitable for this case? If so, how can I revise the code below?

library(rvest);library(dplyr)
url <- "https://www.yemeksepeti.com/istanbul/burger-kadikoy-caferaga-mah-moda#ors:false"
df <- read_html(url) %>% 
  html_nodes("div.restaurant-display-name") %>% 
  html_text()

Thanks in advance.

Upvotes: 0

Views: 829

Answers (1)

Earl Mascetti
Earl Mascetti

Reputation: 1336

You should use RSelenium.

Below one possible solution:

install.packages("RSelenium")
library(RSelenium)

#Start the web driver 
driver <- rsDriver(browser=c("firefox") port = 4445)

#Say 'I'm a client'    
remote_driver <- driver[["client"]]

#Send to browser the web site address    
remote_driver$navigate("https://www.yemeksepeti.com/istanbul/burger-kadikoy-caferaga-mah-moda#ors:false")

#Find in the css enviroment the body   
scroll_d <- remote_driver$findElement(using = "css", value = "body")

#send to the browser to go to the end of the page
scroll_d$sendKeysToElement(list(key = "end"))

Upvotes: 1

Related Questions