CODEWITHSUNDEEP
rweb-scrapingrvest
Brindle Cruncher
Brindle Cruncher

Reputation: 120

Excluding Nodes RVest

I am scraping blog text using RVest and am struggling to figure out a simple way to exclude specific nodes. The following pulls the text:

AllandSundry_test <- read_html
("http://www.sundrymourning.com/2017/03/03/lets-go-back-to-commenting-on-the-weather/")

testpost <- AllandSundry_test %>% 
html_node("#contentmiddle") %>%
html_text() %>%
as.character()

I want to exclude the two nodes with ID's "contenttitle" and "commentblock". Below, I try excluding just the comments using the tag "commentblock".

 testpost <- AllandSundry_test %>% 
   html_node("#contentmiddle") %>%
   html_node(":not(#commentblock)")
   html_text() %>%
   as.character()

When I run this, the result is simply the date -- all the rest of the text is gone. Any suggestions?

I have spent a lot of time searching for an answer, but I am new to R (and html), so I appreciate your patience if this is something obvious.

Upvotes: 5

Views: 3671

Answers (2)

Jesse Kerr
Jesse Kerr

Reputation: 341

It certainly looks like GGamba solved it for you- however, in my machine, I had to remove the > after #contentmiddle. Therefore, this section was instead:

html_nodes("#contentmiddle:not(#commentblock)")

Best of luck! Jesse

Upvotes: 1

GGamba
GGamba

Reputation: 13680

You were almost there. You should use html_nodes instead of html_node.

html_node retrieves the first element it encounter, while html_nodes returns each matching element in the page as a list.
The toString() function collapse the list of strings into one. 

library(rvest)

AllandSundry_test <- read_html("http://www.sundrymourning.com/2017/03/03/lets-go-back-to-commenting-on-the-weather/")

testpost <- AllandSundry_test %>% 
  html_nodes("#contentmiddle>:not(#commentblock)") %>% 
  html_text %>%
  as.character %>%
  toString

testpost
#> [1] "\n\t\tMar\n\t\t3\n\t, Mar, 3, \n\t\tLet's go back to 
#> commenting on the weather\n\t\t\n\t\t, Let's go back to commenting on 
#> the weather, Let's go back to commenting on the weather, I have just 
#> returned from the grocery store, and I need to get something off my chest. 
#> When did "Got any big plans for the rest of the day?" become 
#> the default small ...<truncated>

You still need to clean up the string a bit.

Upvotes: 4

Related Questions