CODEWITHSUNDEEP
bashfile-processinggnu-parallel
drhodes
drhodes

Reputation: 1009

Splitting command line args with GNU parallel

Using GNU parallel: http://www.gnu.org/software/parallel/

I have a program that takes two arguments, e.g.

$ ./prog file1 file2
$ ./prog file2 file3
...
$ ./prog file23456 file23457

I'm using a script that generates the file name pairs, however this poses a problem because the result of the script is a single string - not a pair. like:

$ ./prog "file1 file2"

GNU parallel seems to have a slew of tricks up its sleeves, I wonder if there's one for splitting text around separators:

$ generate_file_pairs | parallel ./prog ?  
  # where ? is text under consideration, like "file1 file2"

The easy work around is to split the args manually in prog, but I'd like to know if it's possible in GNU parallel.

Upvotes: 49

Views: 16453

Answers (4)

tmck-code
tmck-code

Reputation: 3198

Quite late to the party here, but I bump into this problem fairly often and found a nice easy solution

Before passing the arg list to parallel, just replace all the spaces with newlines. I've found tr to be the fastest for this kind of stuff

Not working

echo "1 2 3 4 5"  | parallel echo --
-- 1 2 3 4 5

Working

echo "1 2 3 4 5" | tr ' ' '\n' | parallel echo --
-- 1
-- 2
-- 3
-- 4
-- 5

Protip: before actually running the parallel command, I do 2 things to check that the arguments have been split correctly.

  1. Prepend echo in front of your bash command. This means that any commands that will eventually be executed will be printed for you to check first
  2. Add a marker in the echo, this checks that the parallel split is actually working

> Note, this works best with small/medium argument lists. If the argument list is very large, probably best to just use a for loop to echo each argument to parallel

Upvotes: 3

Liang
Liang

Reputation: 27

In Parallel's manual, it is said:

If no command is given, the line of input is executed ... GNU parallel can often be used as a substitute for xargs or cat | bash.

So take a try of:

generate command | parallel

Try to understand the output of this:

for i in {1..5};do echo "echo $i";done | parallel

Upvotes: 3

Ole Tange
Ole Tange

Reputation: 33685

You are probably looking for --colsep.

generate_file_pairs | parallel --colsep ' ' ./prog {1} {2}

Read man parallel for more. And watch the intro video if you have not already done so http://www.youtube.com/watch?v=OpaiGYxkSuQ

Upvotes: 89

ssapkota
ssapkota

Reputation: 3302

You are looking for -n option of parallel. This is what you are looking for:

./generate_file_pairs | parallel -n 2 ./prog {}

Excerpt from GNU Parallel Doc:

-n max-args
    Use at most max-args arguments per command line. Fewer than max-args 
    arguments will be used if the size (see the -s option) is exceeded, 
    unless the -x option is given, in which case GNU parallel will exit.

Upvotes: 2

Related Questions