CODEWITHSUNDEEP
bashshellunixxargs
joeyhoer
joeyhoer

Reputation: 3767

Monitoring progress within xargs

Is there any way to track or monitor progress within xargs?

I'm replacing a for loop with find … | xargs for performance (specifically to run tasks in parallel). I've read that parallel has a progress flag, but I was wondering if there was a way to accomplish this strictly using xargs.

I'm also aware that tasks run in parallel via xargs will not necessarily finish in the correct order, which compounds the complexity of monitoring progress. Even if a solution would give me a general idea of the progress, that would be a great start.

Upvotes: 4

Views: 4341

Answers (3)

Zaheer
Zaheer

Reputation: 2872

If you're looking for just a general indication of progress the simplest method is to just echo prior to doing the command you'd like to do.

Example: cat <someinput> | xargs -I{} sh -c 'echo {}; <somecmd>;'

-I{} sets {} to the current string being processed

sh -c will allow you to execute multiple commands (note: semi-colon after every command is required including the last one.

Upvotes: 2

Neuron
Neuron

Reputation: 361

If you just want to input how many lines you roughly already processed you may create simple shell function to do that

#!/bin/bash

#-----
##
## @function count
##
## @desc Write every n'th number (if n is 5 write 5, 10, 15, ...)
##
## @param $1 - number 
##
#-----
function count {
   typeset C=0
   while read L; do
      C=$(( C + 1 ))
      if [ $(( $C % $1 )) -eq 0 ]; then
         echo $C 1>&2
      fi
      echo "$L"
   done
}

find . | count 100 | xargs ...

Small problem is that this prints number of lines passed to xargs, not number of lines already processed by the command invoked by xargs. Plus every pipe has some buffer, so it will show slightly higher number than it should. On my machine it showed ~500 lines in advance to real state but if you are processing sufficiently large number of lines 500 is negligible :)

Upvotes: 4

Adrian Fr&#252;hwirth
Adrian Fr&#252;hwirth

Reputation: 45626

You could use pv to monitor the pipe (not in terms of files though, since it sits in your pipe) but you would have to tell pv the size of the find output, which seems really cumbersome (see this answer).

I would really recommend using parallel, based on what you are trying to accomplish...this is exactly what it was designed for.

Otherwise, if you have the Apple Developer Tools installed, you could write a small script that generates a Makefile to accomplish the same task and make sure it prints the progress in a way that you desire.

Upvotes: 0

Related Questions