CODEWITHSUNDEEP
rloopsweb-scrapingrvest
abenol
abenol

Reputation: 91

webscraping loop over url and html node in R rvest

I have a dataframe pubs with two columns: url, html.node. I want to write a loop that reads each url retrieves the html contents, and extract the information indicated by html.node column, and accumulates it in a data frame, or list.
All URL's are different, all html nodes are different.
My code so far is:

score <- vector()
k <- 1
for (r in 1:nrow(pubs)){
  art.url <- pubs[r, 1] # column 1 contains URL
  art.node <- pubs[r, 2] # column 2 contains html nodes as charcters

  art.contents <- read_html(art.url)
  score <- art.contents %>% html_nodes(art.node) %>% html_text()
  k<-k+1
  print(score)
}

I appreciate your help.

Upvotes: 2

Views: 1252

Answers (2)

ha-pu
ha-pu

Reputation: 581

You could also use map from the purrr package instead of a loop:

expand.grid(c("http://quotes.toscrape.com/", "http://quotes.toscrape.com/tag/inspirational/"), # vector of urls
  c(".text",".author"), # vector of nodes
  stringsAsFactors = FALSE) %>% # assuming that the same nodes are relevant for all urls, otherwise you would have to do something like join
  as_tibble() %>%
  set_names(c("url", "node")) %>%
  mutate(out = map2(url, node, ~ read_html(.x) %>% html_nodes(.y) %>% html_text())) %>%
  unnest()

Upvotes: 1

s__
s__

Reputation: 9525

First of all, be sure that each site you're going to scrape, allows you to scrape data, you can incurr in legal issue if you break some rules.

(Note, I've used only http://toscrape.com/ , a sandbox site to scraping, due you did not provide your data)

After that, you can proceed with this, hope it helps:

# first, your data I think they're similar to this
pubs <- data.frame(site = c("http://quotes.toscrape.com/",
                            "http://quotes.toscrape.com/"),
                   html.node = c(".text",".author"), stringsAsFactors = F)

Then the loop you required:

library(rvest)
# an empty list, to fill with the scraped data
empty_list <- list()

# here you are going to fill the list with the scraped data
for (i in 1:nrow(pubs)){
  art.url <- pubs[i, 1]   # choose the site as you did
  art.node <- pubs[i, 2]  # choose the node as you did      
  # scrape it!    
  empty_list[[i]] <- read_html(art.url)  %>% html_nodes(art.node) %>% html_text()

}

Now the result is a list, but, with:

names(empty_list) <- pubs$site

You are going to add to each element of the list the name of the site, with the result:

$`http://quotes.toscrape.com/`
 [1] "“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”"                
 [2] "“It is our choices, Harry, that show what we truly are, far more than our abilities.”"                                              
 [3] "“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”"
 [4] "“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”"                           
 [5] "“Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.”"                    
 [6] "“Try not to become a man of success. Rather become a man of value.”"                                                                
 [7] "“It is better to be hated for what you are than to be loved for what you are not.”"                                                 
 [8] "“I have not failed. I've just found 10,000 ways that won't work.”"                                                                  
 [9] "“A woman is like a tea bag; you never know how strong it is until it's in hot water.”"                                              
[10] "“A day without sunshine is like, you know, night.”"                                                                                 

$`http://quotes.toscrape.com/`
 [1] "Albert Einstein"   "J.K. Rowling"      "Albert Einstein"   "Jane Austen"       "Marilyn Monroe"    "Albert Einstein"   "André Gide"       
 [8] "Thomas A. Edison"  "Eleanor Roosevelt" "Steve Martin"

Clearly it should work with different sites, and different nodes.

Upvotes: 3

Related Questions