CODEWITHSUNDEEP
rrvestrselenium
Nitin Kansal
Nitin Kansal

Reputation: 49

Scroll through a weblink by clicking on "Show More Products" and grab product information

I am able to scrape till 11 scrollings as after that we need to click "Show More Products" button. What can be done so that I can scroll further? Below is my current code:

library(RSelenium)
require(RCurl)
require(XML)
require(dplyr)
require(stringr)
require(rvest)

shell.exec(paste0("C:/Users/Nitin Kansal/Desktop/R/batch.bat"))

#start RSelenium

checkForServer()
startServer()
remDr <- remoteDriver()
remDr$open()

# load your page

remDr$navigate("http://www.jabong.com/kids/clothing/girls-clothing/kids-tops-t-shirts/?source=topnav_kids")

# scroll down 11 times, allowing 3 second for the page to load everytime

for(i in 1:11){      
  remDr$executeScript(paste("scroll(0,",i*10000,");"))
  Sys.sleep(3)    
}

# get the page html

page_source <- remDr$getPageSource()

# get the URL's that you are looking for

pp <- xml2::read_html(page_source[[1]]) %>% 
  rvest::html_nodes("a") %>% 
  rvest::html_attr("data-original-href") %>% 
  {.[!is.na(.)]}

pp <- as.data.frame(pp)

Upvotes: 1

Views: 203

Answers (1)

Bharath
Bharath

Reputation: 1618

After scrolling 11 times try to include this in your code

remDr$findElement(using = 'css selector', ".load-more-products")$clickElement()

That would generate another page and then scrape relevant details.

Loop through the process.

Upvotes: 1

Related Questions