CODEWITHSUNDEEP
rweb-scrapingdownload
Phillip Black
Phillip Black

Reputation: 105

Scraping a series of URLs in R

I'm trying to batch download a series of URLs. So far my code is

link <- paste("http://portal.hud.gov/hudportal/documents/huddoc? id=RAD_PHAApp_", state, ".xls", sep = "")
state <- c('al','tx')
download.file(link, paste(destfile = 'Y:\\PBlack\\RAD\\', state, '.xls', sep = ""), mode = 'wb')

The idea here that I could add names to the state value and it would download and name them the state.

R returns the following when I run the code.

Warning messages:
1: In download.file(link, paste(destfile = "Y:\\PBlack\\RAD\\", state,  :
 only first element of 'url' argument used
2: In download.file(link, paste(destfile = "Y:\\PBlack\\RAD\\", state,  :
 only first element of 'destfile' argument used

Upvotes: 0

Views: 311

Answers (2)

Caroline
Caroline

Reputation: 470

As TARehman points out, you need to have separate calls to download.file for each file. It might be more intuitive for you to do this with a for loop.

Also, using paste0 will avoid using the sep="" every time.

states <- c('al', 'tx')

for(state in states) {
  link <- paste0("http://portal.hud.gov/hudportal/documents/huddoc?id=RAD_PHAApp_", state, ".xls")
  download.file(link,paste0("~/Desktop/",state,".xls"),mode='wb')
}

Though mapply might be somewhat faster.

Upvotes: 0

TARehman
TARehman

Reputation: 6749

You need to call the command to save the files more than once. Right now it will not work because the download.file() function downloads a single file, not a vector of files.

As so:

states <- c('al','tx')
links <- paste("http://portal.hud.gov/hudportal/documents/huddoc?id=RAD_PHAApp_", states, ".xls", sep = "")

func.download_files <- function(link,state) {

    download.file(link,paste("~/Desktop/",state,".xls",sep=""),mode='wb')
}

mapply(FUN = func.download_files,link=links,state=states)

Upvotes: 1

Related Questions