CODEWITHSUNDEEP
bashgzipxargspv
Michel Hua
Michel Hua

Reputation: 1777

gzip on multiple cores with pv progression bar

Following this 2010 question Gzip with all cores, I would like to gzip files using multiple core and indicate a progress bar with pv tool.

How do I improve this code?

CORES=$(grep -c '^processor' /proc/cpuinfo)
find /source -type f -print0 | xargs -0 -n 1 -P $CORES gzip -9

I would like to show remaining time and show the progression bars running in parallel.

Do you have any other best alternatives as of 2018?

Thanks.

Upvotes: 1

Views: 1973

Answers (2)

Mark Setchell
Mark Setchell

Reputation: 207670

Use GNU Parallel which has a progress bar or an eta:

find ... -print0 | parallel -0 --progress gzip -9 {}

Or

find ... -print0 | parallel -0 --eta ...

Or

find ... -print0 | parallel -0 --bar ...

Upvotes: 4

Evert
Evert

Reputation: 99687

You're not really running gzip on multiple cores, you're running multiple gzip's.

pv runs on a pipe. I don't believe it's possible to have it run on multiple pipes at the same time and provide a summary of all of them.

Upvotes: 0

Related Questions