CODEWITHSUNDEEP
rdownload
nate-m
nate-m

Reputation: 557

Use href and target in download.file R?

I have a snippet of code:

raw_prefix <- file.path("data", "raw")

fpa_prefix <- file.path(raw_prefix, "fpa-fod")

if(!dir.exists(fpa_prefix)){
  dir.create(fpa_prefix)
}

fpa_gdb <- file.path(fpa_prefix, "RDS-2013-0009.4_GDB", "Data", "FPA_FOD_20170508.gdb")

if (!file.exists(fpa_gdb)) {
  loc <- "https://www.fs.usda.gov/rds/fedora/objects/RDS:RDS-2013-0009.4/datastreams/RDS-2013-0009.4_GDB/content"
  dest <- paste0(fpa_prefix, ".zip")
  download.file(loc, dest)
  unzip(dest, exdir = fpa_prefix)
  unlink(dest)
  assert_that(file.exists(fpa_gdb))
}

Which works great with most websites to download files on the fly in the name of reproducible workflows, but there is one dataset that I need which has an "href" and "target" file making it very difficult to download using download.file().

The file is found (also in above code) here:

https://www.fs.usda.gov/rds/archive/Product/RDS-2013-0009.4/

Towards the bottom of the page is a file called

RDS-2013-0009.4_GDB.zip

which is the file I am trying to download using the above script.

If you inspect this element you will find this structure, which returns the correct file! But how to translate into R code?

<a href="//www.fs.usda.gov/rds/fedora/objects/RDS:RDS-2013-0009.4/datastreams/RDS-2013-0009.4_GDB/content" target="_blank">RDS-2013-0009.4_GDB.zip</a>

If anyone has an idea on how to download this file I would GREATLY appreciate it!

Thanks!

Upvotes: 1

Views: 529

Answers (1)

hrbrmstr
hrbrmstr

Reputation: 78792

This will:

  • find all the .zip links on the page (URLs and filenames)
  • go through each found and download them "like a browser would do"

Note that write_disk() won't overwrite existing files, so if downloads get interrupted, either delete the file or use overwrite=TRUE.

library(rvest)
library(httr)
library(purrr)

pg <- read_html("https://www.fs.usda.gov/rds/archive/Product/RDS-2013-0009.4/")

fils <- html_nodes(pg, xpath=".//dd[@class='product']//li/a[contains(., 'zip')]") 

walk2(html_attr(fils, 'href'),  html_text(fils), 
      ~GET(sprintf("https:%s", .x), write_disk(.y), progress()))

If you don't want to use purrr, this is all base R:

invisible(
  mapply(
    download.file, 
       url = sprintf("https:%s", html_attr(fils, 'href')),
       destfile = html_text(fils)
  )
)

Upvotes: 3

Related Questions