CODEWITHSUNDEEP
rrvest
youraz
youraz

Reputation: 483

How to scrape multiple pages with an unchanging url in R?

That is the url

My goal is to scrape the review section. But the url doesn't change. Code is given below:

url <- "https://www.n11.com/magaza/thbilisim/magaza-yorumlari"

getreviews <- function(master_df){
  as.data.frame(
    read_html(master_df) %>% 
      html_nodes("div.commentContainer p") %>% 
      html_text()
  )
}

reviews <- url %>% 
  map(getreviews) %>%  
  bind_rows()

How to scrape multiple pages with the same url? Thanks in advance.

Upvotes: 0

Views: 539

Answers (1)

user2474226
user2474226

Reputation: 1502

If you are on a Chrome browser, for example, you can figure out the requested URL per page by going to the Chrome development tools (press F12) and looking at the Network pane.

In your example above, you will see that for every page, the requested URL is https://www.n11.com/component/render/sellerShopFeedbacks?page=page number&sellerId=2145005, where page number is 1, 2, 3, ...

The requested URL pops up on the Network tab when you click on the relevant page number at the bottom of the original URL.

So you just need to increment the page number in your R code to see the subsequent pages.

Upvotes: 2

Related Questions