CODEWITHSUNDEEP
rrvest
Ari
Ari

Reputation: 1972

Scraping a complex HTML table into a data.frame in R

I am trying to load wikipedia's data on US Supreme Court Justices into R:

library(rvest)

html = html("http://en.wikipedia.org/wiki/List_of_Justices_of_the_Supreme_Court_of_the_United_States")
judges = html_table(html_nodes(html, "table")[[2]])
head(judges[,2])

[1] "Wilson, JamesJames Wilson"       "Jay, JohnJohn Jay†"             
[3] "Cushing, WilliamWilliam Cushing" "Blair, JohnJohn Blair, Jr."     
[5] "Rutledge, JohnJohn Rutledge"     "Iredell, JamesJames Iredell"

The problem is that the data is malformed. Rather than the name appearing how I see it in the actual HTML table ("James Wilson"), it is actually appearing twice, once as "Lastname, Firstname" and then once again as "Firstname Lastname".

The reason is that each actually contains an invisible :

<td style="text-align:left;" class="">
    <span style="display:none" class="">Wilson, James</span>
    <a href="/wiki/James_Wilson" title="James Wilson">James Wilson</a>
</td>

The same is also true for the columns with numeric data. I am guessing that this extra code is necessary for sorting the HTML table. However, I am unclear how to remove those spans when trying to create a data.frame from the table in R.

Upvotes: 6

Views: 4295

Answers (2)

lukeA
lukeA

Reputation: 54237

Maybe like this

library(XML)
library(rvest)
html = html("http://en.wikipedia.org/wiki/List_of_Justices_of_the_Supreme_Court_of_the_United_States")
judges = html_table(html_nodes(html, "table")[[2]])
head(judges[,2])
# [1] "Wilson, JamesJames Wilson"       "Jay, JohnJohn Jay†"              "Cushing, WilliamWilliam Cushing" "Blair, JohnJohn Blair, Jr."     
# [5] "Rutledge, JohnJohn Rutledge"     "Iredell, JamesJames Iredel

removeNodes(getNodeSet(html, "//table/tr/td[2]/span"))
judges = html_table(html_nodes(html, "table")[[2]])
head(judges[,2])
# [1] "James Wilson"    "John Jay†"       "William Cushing" "John Blair, Jr." "John Rutledge"   "James Iredell"

Upvotes: 9

Sebastian Barfort
Sebastian Barfort

Reputation: 117

You could use rvest

library(rvest)

html("http://en.wikipedia.org/wiki/List_of_Justices_of_the_Supreme_Court_of_the_United_States")%>%   
  html_nodes("span+ a") %>% 
  html_text()

It's not perfect so you might want to refine the css selector but it gets you fairly close.

Upvotes: 4

Related Questions