CODEWITHSUNDEEP
rxmlrvestrcurl
Sera
Sera

Reputation: 75

Web scraping in R, extracting urls from website with lazy-loading pages

I am trying to extract urls from the website below. The tricky thing here is that the website automatically loads new pages. I did not manage to get the xpath for scraping all urls, including those on the newly loaded pages - I only manage to get the first 15 urls (of more than 70). I assume the xpath in the last line (new_results...) is missing some crucial element to account also for the pages after. Any ideas? Thank you!

# load packages
library(rvest)
library(httr)
library(RCurl)
library(XML)
library(stringr)
library(xml2)


# aim: download all speeches stored at:
# https://sheikhmohammed.ae/en-us/Speeches

# first, create vector which stores all urls to each single speech
all_links <- character() 
new_results <- "/en-us/Speeches"
signatures = system.file("CurlSSL", cainfo = "cacert.pem", package =  "RCurl") 
options(RCurlOptions = list(verbose = FALSE, capath =  system.file("CurlSSL", "cacert.pem", package = "RCurl"), ssl.verifypeer = FALSE))

while(length(new_results) > 0){ 
new_results <- str_c("https://sheikhmohammed.ae", new_results)
results <- getURL(new_results, cainfo = signatures) 
results_tree <- htmlParse(results) 
all_links <- c(all_links, xpathSApply(results_tree,"//div[@class='speech-share-board']", xmlGetAttr,"data-url"))
new_results <- xpathSApply(results_tree,"//div[@class='speech-share-board']//after",xmlGetAttr, "data-url")}

# or, alternatively with phantomjs (also here, it loads only first 15 urls):
url <- "https://sheikhmohammed.ae/en-us/Speeches#"

# write out a script phantomjs can process
writeLines(sprintf("var page = require('webpage').create();
               page.open('%s', function () {
               console.log(page.content); //page source
               phantom.exit();
               });", url), con="scrape.js")

# process it with phantomjs
write(readLines(pipe("phantomjs scrape.js", "r")), "scrape.html")

Upvotes: 0

Views: 1119

Answers (1)

Sera
Sera

Reputation: 75

Running the Javascript for lazy loading in RSelenium or Selenium in Python would be the most elegant approach to solve the problem. Yet, as a less elegant but faster alternative, one can manually change the settings of the json query in the firefox development modus/network feature to load not only 15 but more (=all) speeches at once. This worked fine for me and I was able to extract all the links via the json response.

Upvotes: 1

Related Questions