CODEWITHSUNDEEP
rdplyrrvest
Nettle
Nettle

Reputation: 3321

Scraping Hyperlinks with Rvest

I'd like to scrape the text and hyperlinks (of .xlsx and .pdf files) from a page using rvest. I'm not very good at this, so it's hard to tell if I'm dealing with a complicated webpage, or am just making newbie mistakes. My code thus far:

my.url <- "https://comptroller.defense.gov/Budget-Materials/Budget2019/"
my.xpath <- '//*[@id="LiveHTMLWrapper92093"]/div/div'

x <- read_html(my.url) %>% 
  html_node(xpath = my.xpath) 

{xml_node}
<div style="width: 710px; height: 600px; overflow: auto;">
[1] <h5 style="text-align: left; background-color: #dbdbe4;"><a name="press" style=" ...
[2] <p><a href="/Portals/45/Documents/defbudget/fy2019/fy2019_Press_Release.pdf" sty ...
[3] <p style="margin-top: 1px; margin-bottom: 0px;"><strong><span style="font-family ...
[4] <p style="margin-top: 1px; margin-bottom: 0px;"><strong><span style="font-family ...
[5] <p><strong>\n- <a href="https://www.defense.gov/News/Transcripts/Transcript-View ...
[6] <h5 style="text-align: left; background-color: #dbdbe4;"><a name="summary" style ...
[7] <div style="height: 50px;">\n<a href="/Portals/45/Documents/defbudget/fy2019/FY2 ...
[8] <strong><strong>\n<b><strong>\n<b>\n<strong>\n</strong>\n<strong>\n</strong>\n<s ...

Ideally, I'd like to output a dataframe containing the text in one column and associated href in another.

Upvotes: 1

Views: 241

Answers (1)

Alexandre georges
Alexandre georges

Reputation: 667

Here a solution :

my.url <- "https://comptroller.defense.gov/Budget-Materials/Budget2019/"
my.xpath <- '//*[@id="dnn_ctr92093_ContentPane"]'

x <- read_html(my.url) %>% 
  html_node(xpath = my.xpath) %>% html_nodes("a") %>% html_text()

y <- read_html(my.url) %>% 
  html_node(xpath = my.xpath) %>% html_nodes("a") %>% html_attr("href") 

y <- ifelse(grepl(pattern = "/Portals/",y), paste0("https://comptroller.defense.gov",y),y)

df <- as.data.frame(cbind(x,y))

Upvotes: 1

Related Questions