Download files when exact URL is not known

Question

From this link, I´m trying to download multiple pdf files, but I can´t get the exact URL for each file.

To access one of the pdf files, you could click on "Región de Arica y Parinacota" and then click on "Arica". Then, you can check that the url is http://cdn.servel.cl/padronesauditados/padron/A1501001.pdf, if you click on the next link "Camarones" you now noticed that the URL is http://cdn.servel.cl/padronesauditados/padron/A1501002.pdf

I checked more URLs, and they all have a similar pattern:

"A" + "two digit number from 1 to 15" + "two digit number of unknown range" + "three digit number of unknown range"

Even though the URL examples I showed seem to suggest that the file names are named sequentally, this is not always the case.

What I did to be able to download all the files despite not knowing the exact URLs I did the following:

1) I made a for loop in order to write all possible file names based on the pattern I describe above, i.e, A0101001.pdf, A0101002.pdf....A1599999.pdf

library(downloader)
library(stringr)
reg.ind <- 1:15
pro.ind <- 1:99
com.ind <- 1:999
reg <- str_pad(reg.ind, width=2, side="left", pad="0")
prov <- str_pad(pro.ind, width=2, side="left", pad="0")
com  <- str_pad(com.ind, width=3, side="left", pad="0")

file <- c()
for(i in 1:length(reg)){
 reg.i <- reg[i]
    for(j in 1:length(prov)){
        prov.j <- prov[j]
            for(k in 1:length(com)){
                com.k <- com[k]
                file <- c(file, (paste0("A", reg.i, prov.j, com.k)))
            }
        }
    }

2) then I used another for loop to download a file everytime I hit a correct URL. I use tryCatchto ignore the cases when the URL was incorrect (most of the time)

for(i in 1:length(file)){
 tryCatch({
 url <- paste0("http://cdn.servel.cl/padronesauditados/padron/", file[i], 
 ".pdf")
# change destfile accordingly if you decide to run the code
download.file(url, destfile = paste0("./datos/comunas/",  file[i], ".pdf"), 
mode = "wb")
}, error = function(e){})
}

PROBLEM: In total I know there are not more than 400 pdf files, as each one of them correspond to a commune in Chile, but I wrote a vector with 1483515 possible file names, and therefore my code, even though it works, takes a much longer time than if I could manage to obtain the file names before hand.

Does anyone know how to workaround this problem?

Download files when exact URL is not known

Answers (1)

Related Questions