CODEWITHSUNDEEP
htmlrweb-scrapingrselenium
indiemathematician
indiemathematician

Reputation: 19

Dynamic Scraping in R: Webpages that require user to scroll to load more information

I am trying to write a a single RScript that will scrape college ranking information from https://www.usnews.com/best-colleges/rankings/regional-universities-north. This site requires scrolling down to load more college information.

My hope is to do this without the use of docker or an outside program that requires command line prompts. From my attempts, it seems that RSelenium is defunct as well.

Upvotes: 1

Views: 66

Answers (1)

Nad Pat
Nad Pat

Reputation: 3173

Here is a partial answer,

library(RSelenium) 
driver <- rsDriver(browser=c("firefox"))
remDr <- driver$client

remDr$navigate("https://www.usnews.com/best-colleges/rankings/regional-universities-north")
# Name of the college 
remDr$findElement(using = "xpath",'//*[@id="school-0"]/div/div[1]/div[1]/div[1]/h3') -> collegename
collegename$getElementText()
#Rank of the college 
remDr$findElement(using = "xpath",'//*[@id="school-0"]/div/div[1]/div[1]/div[2]/ul/li/a/div') -> rankcollege
rankcollege$getElementText()

This gives you college name and rank.

But to get rank for all colleges you have to loop. To scroll down the page refer, Scrolling page in RSelenium

Upvotes: 1

Related Questions