CODEWITHSUNDEEP
rweb-scrapingrvest
M M
M M

Reputation: 21

web scraping with R, clicking on links

I am a beginner, I would like to scrape all the articles with the selected keyword from the page. I can scrape only the titles of articles displayed on a single page, a part of the description of articles and their links. I would not only like to scrape the search results, but also the content of each of the displayed links.

website: http://search.time.com/?site=time&q=bitcoin 

require(rvest)
url<- "http://search.time.com/?site=time&q=bitcoin"
webpage <- read_html(url)

title_data_html <- html_nodes(webpage,'.content-title a')

title_data <- html_text(title_data_html)

description_data_html <- html_nodes(webpage,'.content-snippet')
description_data <- html_text(description_data_html)

links = html_attr(title_data_html, name = "href")

Upvotes: 2

Views: 2330

Answers (1)

Amar
Amar

Reputation: 1373

The function you're after is follow_link() from the rvest package. Here is another SO post about this topic:

Scraping linked HTML webpages by looping the rvest::follow_link() function

Upvotes: 1

Related Questions