CODEWITHSUNDEEP
unixlz4
Cdr
Cdr

Reputation: 581

Decompressing multiple files using LZ4

I have a set of compressed lz4 files(40), which is around 50GB. Need to de-compress that. Using following code to decompress.

lz4 -d -m fileA fileB fileC fileD....

It's taking hours to decompress in 2 sockets machine with 32 cores and 256gb RAM. But it's using only single core. As per documentation, lz4 is multithreaded.

So i divided the task as 4 different tasks

nohup lz4 -d fileA fileB &
nohup lz4 -d fileC fileD &
nohup lz4 -d fileE fileF &
nohup lz4 -d fileG fileH &

But still, it's taking 1.5 hrs to complete. Total decompressed data is around 1.1TB.

What would be the best approach to attain maximum performance?

Upvotes: 0

Views: 842

Answers (0)

Related Questions