CODEWITHSUNDEEP
rcsvgziptarin-memory
imriss
imriss

Reputation: 1971

How to directly perform write.csv in R into tar.gz format?

I have a big data.frame that I want to write into a compressed CSV file. Is there any way to directly write the data into a CSV.TAR.GZ compressed file instead of performing write.csv/gzip steps in order to reduce DISK access?

Thanks.

Upvotes: 24

Views: 13491

Answers (2)

Ajay
Ajay

Reputation: 454

For big data.frames (or large files) fwrite is much faster than write.csv. For example

data.table::fwrite(df, file = "file.csv.gz")

command took 59.4 sec while

write.csv(df, file=gzfile("file.csv.gz"))

took 18.1 mins to write the same data to a 0.8GB file.

Upvotes: 1

Hong Ooi
Hong Ooi

Reputation: 57686

Use gzfile (or bzfile for bzip2 archiving, or xzfile for xz archiving).

write.csv(mtcars, file=gzfile("mtcars.csv.gz"))

PS. If you only have one data frame, surely you don't need tar.

Upvotes: 51

Related Questions