CODEWITHSUNDEEP
htmlrmacosweb-scrapingrvest
lavenderGem
lavenderGem

Reputation: 117

rvest html_nodes() returns empty character

I am trying to scrape a website (https://genelab-data.ndc.nasa.gov/genelab/projects?page=1&paginate_by=281). In particular, I am trying to scrape all 281 "release dates" (with the first being '30-Oct-2006')

To do this, I am using the R package rvest and the SelectorGadget Chrome extension. I am using Mac version 10.15.6.

I attempted the following code:

library(rvest)
library(httr)
library(xml2)
library(dplyr)

link = "https://genelab-data.ndc.nasa.gov/genelab/projects?page=1&paginate_by=281"
page = read_html(link)    
year = page %>% html_nodes("td:nth-child(4) ul") %>% html_text()

However, this returns 'character(0)`.

I used the code td:nth-child(4) ul because this is what SelectorGadget highlighted for each of the 281 release dates. I also tried to "View source page" but could not find these years listed on the source page.

I have read that rvest does not always work depending on the type of website. In this case, what is a possible workaround? Thank you.

Upvotes: 1

Views: 500

Answers (1)

Bertrand Martel
Bertrand Martel

Reputation: 45402

This site gets the data from this API call https://genelab-data.ndc.nasa.gov/genelab/data/study/all that returns JSON data. You can use httr to get the data and parse JSON :

library(httr)

url <- "https://genelab-data.ndc.nasa.gov/genelab/data/study/all"

output <- content(GET(url), as = "parsed", type = "application/json")

#sort by glds_id
output = output[order(sapply(output, `[[`, i = "glds_id"))]

#build dataframe
result <- list();
index <- 1
for(t in output[length(output):1]){
    result[[index]] <- t$metadata
    result[[index]]$accession <- t$accession
    result[[index]]$legacy_accession <- t$legacy_accession
    index <- index + 1
}

df <- do.call(rbind, result)
options(width = 1200)
print(df)

Output sample (without all columns)

       accession legacy_accession public_release_date title                                                                                                            
  [1,] "GLDS329" "GLDS-329"       "30-Oct-2006"       "Transcription profiling of atm mutant, adm mutant and wild type whole plants and roots of Arabidops" [truncated]
  [2,] "GLDS322" "GLDS-322"       "27-Aug-2020"       "Comparative RNA-Seq transcriptome analyses reveal dynamic time dependent effects of 56Fe, 16O, and " [truncated]
  [3,] "GLDS320" "GLDS-320"       "18-Sep-2014"       "Gamma radiation and HZE treatment of seedlings in Arabidopsis"                                                  
  [4,] "GLDS319" "GLDS-319"       "18-Jul-2018"       "Muscle atrophy, osteoporosis prevention in hibernating mammals"                                                 
  [5,] "GLDS318" "GLDS-318"       "01-Dec-2019"       "RNA seq of tumors derived from irradiated versus sham hosts transplanted with Trp53 null mammary ti" [truncated]
  [6,] "GLDS317" "GLDS-317"       "19-Dec-2017"       "Galactic cosmic radiation induces stable epigenome alterations relevant to human lung cancer"                   
  [7,] "GLDS311" "GLDS-311"       "31-Jul-2020"       "Part two: ISS Enterobacteriales"                                                                                
  [8,] "GLDS309" "GLDS-309"       "12-Aug-2020"       "Comparative Genomic Analysis of Klebsiella Exposed to Various Space Conditions at the International" [truncated]
  [9,] "GLDS308" "GLDS-308"       "07-Aug-2020"       "Differential expression profiles of long non-coding RNAs during the mouse pronucleus stage under no" [truncated]
 [10,] "GLDS305" "GLDS-305"       "27-Aug-2020"       "Transcriptomic responses of Serratia liquefaciens cells grown under simulated Martian conditions of" [truncated]
 [11,] "GLDS304" "GLDS-304"       "28-Aug-2020"       "Global gene expression in response to X rays in mice deficient in Parp1"                                        
 [12,] "GLDS303" "GLDS-303"       "15-Jun-2020"       "ISS Bacillus Genomes"                                                                                           
 [13,] "GLDS302" "GLDS-302"       "31-May-2020"       "ISS Enterobacteriales Genomes"                                                                                  
 [14,] "GLDS301" "GLDS-301"       "30-Apr-2020"       "Eruca sativa Rocket Science RNA-seq"                                                                            
 [15,] "GLDS298" "GLDS-298"       "09-May-2020"       "Draft Genome Sequences of Sphingomonas sp. Isolated from the International Space Station Genome seq" [truncated]
 ...........................................................................

Upvotes: 1

Related Questions