CODEWITHSUNDEEP
rcompression
João Daniel
João Daniel

Reputation: 8986

Automate zip file reading in R

I need to automate R to read a csv datafile that's into a zip file.

For example, I would type:

read.zip(file = "myfile.zip")

And internally, what would be done is:

If there is more than one file into the zip file, an error is thrown.

My problem is to get the name of the file contained into the zip file, in orded to provide it do the read.csv command. Does anyone know how to do it?

UPDATE

Here's the function I wrote based on @Paul answer:

read.zip <- function(zipfile, row.names=NULL, dec=".") {
    # Create a name for the dir where we'll unzip
    zipdir <- tempfile()
    # Create the dir using that name
    dir.create(zipdir)
    # Unzip the file into the dir
    unzip(zipfile, exdir=zipdir)
    # Get the files into the dir
    files <- list.files(zipdir)
    # Throw an error if there's more than one
    if(length(files)>1) stop("More than one data file inside zip")
    # Get the full name of the file
    file <- paste(zipdir, files[1], sep="/")
    # Read the file
    read.csv(file, row.names, dec)
}

Since I'll be working with more files inside the tempdir(), I created a new dir inside it, so I don't get confused with the files. I hope it may be useful!

Upvotes: 24

Views: 20922

Answers (9)

Gucci148
Gucci148

Reputation: 2157

unzipped file location

outDir<-"~/Documents/unzipFolder"

get all the zip files

zipF <- list.files(path = "~/Documents/", pattern = "*.zip", full.names = TRUE)

unzip all your files

purrr::map(.x = zipF, .f = unzip, exdir = outDir)

Upvotes: 2

altabq
altabq

Reputation: 1424

Another approach that uses fread from the data.table package

fread.zip <- function(zipfile, ...) {
  # Function reads data from a zipped csv file
  # Uses fread from the data.table package

  ## Create the temporary directory or flush CSVs if it exists already
  if (!file.exists(tempdir())) {dir.create(tempdir())
  } else {file.remove(list.files(tempdir(), full = T, pattern = "*.csv"))
  }

  ## Unzip the file into the dir
  unzip(zipfile, exdir=tempdir())

  ## Get path to file
  file <- list.files(tempdir(), pattern = "*.csv", full.names = T)

  ## Throw an error if there's more than one
  if(length(file)>1) stop("More than one data file inside zip")

  ## Read the file
  fread(file, 
     na.strings = c(""), # read empty strings as NA
     ...
  )
}

Based on the answer/update by @joão-daniel

Upvotes: 1

rbatt
rbatt

Reputation: 4807

Here is an approach I am using that is based heavily on @Corned Beef Hash Map 's answer. Here are some of the changes I made:

  • My approach makes use of the data.table package's fread(), which can be fast (generally, if it's zipped, sizes might be large, so you stand to gain a lot of speed here!).

  • I also adjusted the output format so that it is a named list, where each element of the list is named after the file. For me, this was a very useful addition.

  • Instead of using regular expressions to sift through the files grabbed by list.files, I make use of list.file()'s pattern argument.

  • Finally, I by relying on fread() and by making pattern an argument to which you could supply something like "" or NULL or ".", you can use this to read in many types of data files; in fact, you can read in multiple types of at once (if your .zip contains .csv, .txt in you want both, e.g.). If there are only some types of files you want, you can specify the pattern to only use those, too.

Here is the actual function:

read.csv.zip <- function(zipfile, pattern="\\.csv$", ...){

    # Create a name for the dir where we'll unzip
    zipdir <- tempfile()

    # Create the dir using that name
    dir.create(zipdir)

    # Unzip the file into the dir
    unzip(zipfile, exdir=zipdir)

    # Get a list of csv files in the dir
    files <- list.files(zipdir, rec=TRUE, pattern=pattern)

    # Create a list of the imported csv files
    csv.data <- sapply(files, 
        function(f){
            fp <- file.path(zipdir, f)
            dat <- fread(fp, ...)
            return(dat)
        }
    )

    # Use csv names to name list elements
    names(csv.data) <- basename(files)

    # Return data
    return(csv.data)
}

Upvotes: 2

Victor Trevino
Victor Trevino

Reputation: 1

I just wrote a function based on top read.zip that may help...

read.zip <- function(zipfile, internalfile=NA, read.function=read.delim, verbose=TRUE, ...) {
    # function based on http://stackoverflow.com/questions/8986818/automate-zip-file-reading-in-r

    # check the files within zip
    unzfiles <- unzip(zipfile, list=TRUE)
    if (is.na(internalfile) || is.numeric(internalfile)) {
        internalfile <- unzfiles$Name[ifelse(is.na(internalfile),1,internalfile[1])]
    }
    # Create a name for the dir where we'll unzip
    zipdir <- tempfile()
    # Create the dir using that name
    if (verbose) catf("Directory created:",zipdir,"\n")
    dir.create(zipdir)
    # Unzip the file into the dir
    if (verbose) catf("Unzipping file:",internalfile,"...")
    unzip(zipfile, file=internalfile, exdir=zipdir)
    if (verbose) catf("Done!\n")
    # Get the full name of the file
    file <- paste(zipdir, internalfile, sep="/")
    if (verbose) 
        on.exit({ 
            catf("Done!\nRemoving temporal files:",file,".\n") 
            file.remove(file)
            file.remove(zipdir)
            }) 
    else
        on.exit({file.remove(file); file.remove(zipdir);})
    # Read the file
    if (verbose) catf("Reading File...")
    read.function(file, ...)
}

Upvotes: 0

Jack Wasey
Jack Wasey

Reputation: 3440

The following refines the above answers. FUN could be read.csv, cat, or anything you like, providing the first argument will accept a file path. E.g.

head(read.zip.url("http://www.cms.gov/Medicare/Coding/ICD9ProviderDiagnosticCodes/Downloads/ICD-9-CM-v32-master-descriptions.zip", filename = "CMS32_DESC_LONG_DX.txt"))

read.zip.url <- function(url, filename = NULL, FUN = readLines, ...) {
  zipfile <- tempfile()
  download.file(url = url, destfile = zipfile, quiet = TRUE)
  zipdir <- tempfile()
  dir.create(zipdir)
  unzip(zipfile, exdir = zipdir) # files="" so extract all
  files <- list.files(zipdir)
  if (is.null(filename)) {
    if (length(files) == 1) {
      filename <- files
    } else {
      stop("multiple files in zip, but no filename specified: ", paste(files, collapse = ", "))
    }
  } else { # filename specified
    stopifnot(length(filename) ==1)
    stopifnot(filename %in% files)
  }
  file <- paste(zipdir, files[1], sep="/")
  do.call(FUN, args = c(list(file.path(zipdir, filename)), list(...)))
}

Upvotes: 1

Corned Beef Hash Map
Corned Beef Hash Map

Reputation: 221

I found this thread as I was trying to automate reading multiple csv files from a zip. I adapted the solution to the broader case. I haven't tested it for weird filenames or the like, but this is what worked for me so I thought I'd share:

read.csv.zip <- function(zipfile, ...) {
# Create a name for the dir where we'll unzip
zipdir <- tempfile()
# Create the dir using that name
dir.create(zipdir)
# Unzip the file into the dir
unzip(zipfile, exdir=zipdir)
# Get a list of csv files in the dir
files <- list.files(zipdir)
files <- files[grep("\\.csv$", files)]
# Create a list of the imported csv files
csv.data <- sapply(files, function(f) {
    fp <- file.path(zipdir, f)
    return(read.csv(fp, ...))
})
return(csv.data)}

Upvotes: 4

Holger Brandl
Holger Brandl

Reputation: 11192

If you have zcat installed on your system (which is the case for linux, macos, and cygwin) you could also use:

zipfile<-"test.zip"
myData <- read.delim(pipe(paste("zcat", zipfile)))

This solution also has the advantage that no temporary files are created.

Upvotes: 2

Joshua Ulrich
Joshua Ulrich

Reputation: 176668

Another solution using unz:

read.zip <- function(file, ...) {
  zipFileInfo <- unzip(file, list=TRUE)
  if(nrow(zipFileInfo) > 1)
    stop("More than one data file inside zip")
  else
    read.csv(unz(file, as.character(zipFileInfo$Name)), ...)
}

Upvotes: 11

Paul Hiemstra
Paul Hiemstra

Reputation: 60944

You can use unzip to unzip the file. I just mention this as it is not clear from your question whether you knew that. In regard to reading the file. Once your extracted the file to a temporary dir (?tempdir), just use list.files to find the files that where dumped into the temporary directory. In your case this is just one file, the file you need. Reading it using read.csv is then quite straightforward:

l = list.files(temp_path)
read.csv(l[1])

assuming your tempdir location is stored in temp_path.

Upvotes: 10

Related Questions