CODEWITHSUNDEEP
rdata.tablezip
francoiskroll
francoiskroll

Reputation: 1098

R: data.table fread on zip containing multiple files

What is an OS-agnostic solution to use fread on a zip? I can't seem to find one.

Setting up the stage

Let's create two dataframes, write them to disk, and put them in a zip archive (I stole this from: How to zip multiple CSV files in R?)

library(zip)
df1 <- head(mtcars)
df2 <- head(iris)
write.csv(df1, 'df1.csv')
write.csv(df2, 'df2.csv')
zip(zipfile='df.zip', files=list.files(path = getwd(), pattern = ".csv$"))

Now I want to read this zip into R

Let's say I want to read df1.csv from the zip.

fread('df.zip/df1.csv')

Error in fread("df.zip/df1.csv") : File 'df.zip/df1.csv' does not exist or is non-readable

I tried this from fread() of file from archive

fread('unzip -p df.zip/df1.csv')

Null data.table (0 rows and 0 cols)
Warning message:
In fread("unzip -p df.zip/df1.csv") : File '/var/folders/w5/kqy78qb17v176195dtyyc4pj40000gn/T//RtmpIlNSk8/filee1693cc7f89' has size 0. Returning a NULL data.table.

I do not understand what it is trying to import, but clearly not my dataframe of interest.

Can you help?

Edit 1

Unzipping first is not really an option. In practice, I am working with batches of highly compressible files. Usually ~ 3000 xls files, each 1M rows. 100 Gb unzipped / 8 Gb zipped. Needless to say it would be much more comfortable to read directly from the zip!

Upvotes: 3

Views: 2196

Answers (1)

Waldi
Waldi

Reputation: 41230

Having unzip installed, this solution works on my computer :

fread(cmd = 'unzip -p df.zip df1.csv')
                  V1  mpg cyl disp  hp drat    wt  qsec vs am gear carb
1:         Mazda RX4 21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
2:     Mazda RX4 Wag 21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
3:        Datsun 710 22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
4:    Hornet 4 Drive 21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
5: Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2
6:           Valiant 18.1   6  225 105 2.76 3.460 20.22  1  0    3    1

Upvotes: 5

Related Questions