CODEWITHSUNDEEP
rcurl
anikaM
anikaM

Reputation: 429

How to download .gz files from FTP server in R?

I am trying to download all .gz files from this link: ftp://ftp.ncbi.nih.gov/snp/organisms/human_9606_b151_GRCh38p7/BED/

So far I tried this and I am not getting any results:

require(RCurl)

url= "ftp://ftp.ncbi.nih.gov/snp/organisms/human_9606_b151_GRCh38p7/BED/"
filenames = getURL(url, ftp.use.epsv = FALSE, dirlistonly = TRUE)
filenames <- strsplit(filenames, "\r\n")
filenames = unlist(filenames)

I am getting this error:

Error in function (type, msg, asError = TRUE)  : 
  Operation timed out after 300552 milliseconds with 0 out of 0 bytes  received

Can someone please help with this?

Thanks

EDIT: I tried to run with with filenames provided to me bellow so in my r script I have:

require(RCurl)
my_url <-"ftp://ftp.ncbi.nih.gov/snp/organisms/human_9606_b151_GRCh38p7/BED/"

my_filenames= c("bed_chr_11.bed.gz", ..."bed_chr_9.bed.gz.md5")

my_filenames <- strsplit(my_filenames, "\r\n")
my_filenames = unlist(my_filenames)

for(my_file in my_filenames){
download.file(paste0(my_url, my_file), destfile =  file.path('/mydir', my_file))
}

And when I run the script I get these warnings:

trying URL 'ftp://ftp.ncbi.nih.gov/snp/organisms/human_9606_b151_GRCh38p7/BED/bed_chr_11.bed.gz'

Error in download.file(paste0(my_url, my_file), destfile = file.path("/mydir", : cannot open URL 'ftp://ftp.ncbi.nih.gov/snp/organisms/human_9606_b151_GRCh38p7/BED/bed_chr_11.bed.gz' In addition: Warning message: In download.file(paste0(my_url, my_file), destfile = file.path("/mydir", : URL 'ftp://ftp.ncbi.nih.gov/snp/organisms/human_9606_b151_GRCh38p7/BED/bed_chr_11.bed.gz': status was 'Timeout was reached' Execution halted

Upvotes: 2

Views: 791

Answers (1)

Joseph Clark McIntyre
Joseph Clark McIntyre

Reputation: 1094

The file names you're trying to access are

filenames <- c("bed_chr_11.bed.gz", "bed_chr_11.bed.gz.md5", "bed_chr_12.bed.gz", 
"bed_chr_12.bed.gz.md5", "bed_chr_13.bed.gz", "bed_chr_13.bed.gz.md5", 
"bed_chr_14.bed.gz", "bed_chr_14.bed.gz.md5", "bed_chr_15.bed.gz", 
"bed_chr_15.bed.gz.md5", "bed_chr_16.bed.gz", "bed_chr_16.bed.gz.md5", 
"bed_chr_17.bed.gz", "bed_chr_17.bed.gz.md5", "bed_chr_18.bed.gz", 
"bed_chr_18.bed.gz.md5", "bed_chr_19.bed.gz", "bed_chr_19.bed.gz.md5", 
"bed_chr_20.bed.gz", "bed_chr_20.bed.gz.md5", "bed_chr_21.bed.gz", 
"bed_chr_21.bed.gz.md5", "bed_chr_22.bed.gz", "bed_chr_22.bed.gz.md5", 
"bed_chr_AltOnly.bed.gz", "bed_chr_AltOnly.bed.gz.md5", "bed_chr_MT.bed.gz", 
"bed_chr_MT.bed.gz.md5", "bed_chr_Multi.bed.gz", "bed_chr_Multi.bed.gz.md5", 
"bed_chr_NotOn.bed.gz", "bed_chr_NotOn.bed.gz.md5", "bed_chr_PAR.bed.gz", 
"bed_chr_PAR.bed.gz.md5", "bed_chr_Un.bed.gz", "bed_chr_Un.bed.gz.md5", 
"bed_chr_X.bed.gz", "bed_chr_X.bed.gz.md5", "bed_chr_Y.bed.gz", 
"bed_chr_Y.bed.gz.md5", "bed_chr_1.bed.gz", "bed_chr_1.bed.gz.md5", 
"bed_chr_10.bed.gz", "bed_chr_10.bed.gz.md5", "bed_chr_2.bed.gz", 
"bed_chr_2.bed.gz.md5", "bed_chr_3.bed.gz", "bed_chr_3.bed.gz.md5", 
"bed_chr_4.bed.gz", "bed_chr_4.bed.gz.md5", "bed_chr_5.bed.gz", 
"bed_chr_5.bed.gz.md5", "bed_chr_6.bed.gz", "bed_chr_6.bed.gz.md5", 
"bed_chr_7.bed.gz", "bed_chr_7.bed.gz.md5", "bed_chr_8.bed.gz", 
"bed_chr_8.bed.gz.md5", "bed_chr_9.bed.gz", "bed_chr_9.bed.gz.md5"
)

The files are big, so I didn't check that this whole loop, but this worked at least for the first file. Add this to the end of your code.

my_url <- 'ftp://ftp.ncbi.nih.gov/snp/organisms/human_9606_b151_GRCh38p7/BED/'
for(my_file in filenames){ # loop over the files
  # download each file, saving in a directory that you need to create on your own computer
  download.file(paste0(my_url, my_file), destfile = file.path('c:/users/josep/Documents/', my_file))
}

Upvotes: 1

Related Questions