Using R for web scraping

Question

I need to get the list from a lot of webpages like this: https://fossilplants.info/genus.htm?page=3 I tried to do that using several R packages, such as rvest and XML, but didn't figure out how to make it work. Could anyone help me on that? Thanks so much.

Answer 1

We can use rvest like :

library(rvest)
library(purrr)

url <- 'https://fossilplants.info/genus.htm?page=3'

url %>%
  read_html() %>%
  html_nodes('h1') %>%
  html_text() %>%
  gsub('[\r\n\t]', '', .)

# [1] "Genus Abies-pollenites Thierg. in Raatz Abh. Preuss. Geol. Landesanst., Neue Folge, (183): 16.  26 Jan 1938"               
# [2] "Genus Abieticedripites Maljavk. Trudy Vsesoyuzn. Neft. Nauchno-Issl. Geol.-Razved.  Inst., N. S., (119): 103.  11 Jul 1958"
# [3] "Genus Abietineae-pollenites R. Potonié Palaeontographica, Abt. B, Paläophytol., 91(5-6): 144, 145.  Apr 1951"              
# [4] "Genus Abietineaepollenites R. Potonié in Delcourt, Sprumont Mém. Soc. Belge Géol.,  N. Sér. 4°, (5): 51.  1955"            
# [5] "Genus Abietipites Wodehouse Bull. Torrey Bot. Club, 60(7): 491.  Oct 1933"                                                 
# [6] "Genus Abietites Maljavk. Trudy Vsesoyuzn. Neft. Nauchno-Issl. Geol.-Razved.  Inst., N. S., (231): 142.  10 Aug 1964"       
# [7] "Genus Abietites Hising. Lethaea Svecica 110.  7 Dec 1836"                                                                  
# [8] "Genus Abietopitys Kräusel Beitr. Geol. Erforsch. Deutsch. Schutzgeb., (20): 32.  11 Aug 1928"                              
# [9] "Unranked Abietosaccites Erdtman Svensk Bot. Tidskr., 41(1): 110.  26 Mar 1947"                                             
#[10] "Genus Abietoxylon  73.  " 

If you want to do this for multiple pages, you can change the url like and perform the same function.

map(1:3, ~{
   url <- sprintf('https://fossilplants.info/genus.htm?page=%d', .x)
   url %>%
     read_html() %>%
     html_nodes('h1') %>%
     html_text() %>%
     gsub('[\r\n\t]', '', .)
}) %>% flatten_chr()