CODEWITHSUNDEEP
linuxtar
user984003
user984003

Reputation: 29597

How to limit memory usage during tar

I need to tar (or otherwise zip) more than 2.7 million files. (150GB)

However, with this many files the tar command uses way too much memory and my system crashes. What can I do?

tar -cf /path/filename.tar /path_to_file/

I've tried to do it in batches (multiple tar files would be OK) based on file creation date and find, but find takes up even more memory.

Upvotes: 2

Views: 5229

Answers (1)

steezeburger
steezeburger

Reputation: 570

Not sure if this is an answer exactly, as it doesn't say how to explicitly lower tar's memory usage, but...

I think you can specify the compression program used with tar to use pigz (parallel gzip), then specify the number of threads to use to better help manage memory. Maybe something like:

tar cvf - paths-to-archive | pigz -p 4 > archive.tar.gz

where -p $NUM is the number of cores.

Upvotes: 2

Related Questions