CODEWITHSUNDEEP
r
user113156
user113156

Reputation: 7107

`walk` to download a file and store link url as file name

I am trying to extract some info from a number of links.

I am applying the following function:

walk(filinginfohref, function(x) {
  download.file(x, destfile = paste0("D:/deleteme/",x), quiet = FALSE)
})

However it returns the following error:

 Error in download.file(x, destfile = paste0("D:/deleteme/", x), quiet = FALSE) : 
  cannot open destfile 'D:/deleteme/https://www.sec.gov/Archives/edgar/data/1750/000104746918004978/0001047469-18-004978-index.htm', reason 'Invalid argument'

Which I assume is because I cannot store the link as the destination file.

I need to somehow preserve the link from where the file is being downloaded from

How can I overcome this issue?

Data

filinginfohref <- c("https://www.sec.gov/Archives/edgar/data/1750/000104746918004978/0001047469-18-004978-index.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746917004528/0001047469-17-004528-index.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746916014299/0001047469-16-014299-index.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746915006136/0001047469-15-006136-index.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746914006243/0001047469-14-006243-index.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746913007797/0001047469-13-007797-index.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746912007300/0001047469-12-007300-index.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746911006302/0001047469-11-006302-index.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746910006500/0001047469-10-006500-index.htm", 
"https://www.sec.gov/Archives/edgar/data/1750/000104746909006783/0001047469-09-006783-index.htm"
)

Upvotes: 1

Views: 70

Answers (2)

C&#233;sar Arquero Cabral
C&#233;sar Arquero Cabral

Reputation: 499

As you have figure it out, windows doesn't allow you to save those name files with the special characters. Add a function to remove the common part of the file name and get rid of those "/".

library(purrr)

htmName <- function (x) {
    x <- gsub("https://www.sec.gov/Archives/edgar/data/", "",x)
    x <- gsub("/","_",x)
    return(x)
}

walk(filinginfohref, function(x) {
    download.file(x, destfile = paste0("output/", htmName(x)), quiet = FALSE)
})

Upvotes: 1

Clemsang
Clemsang

Reputation: 5481

Each link have / interpreted as folders. The path that is built does not exist.

Please replace destfile = paste0("D:/deleteme/",x) by destfile = paste0("D:/deleteme/", gsub("/", "_", x, fixed = TRUE))

This way you have the character _ avoiding troubles.

There is probably a way to keep links intacts.

Upvotes: 1

Related Questions