CODEWITHSUNDEEP
parallel-processinggnu-parallel
Omer Dagan
Omer Dagan

Reputation: 15976

GNU parallel does not run in parallel on remote servers when using --onall

I'm trying to run a script on multiple remote servers, with multiple parameters. The GNU parallel command is:

parallel --onall -S ${RH32},{RH64} /shared/loc/script.sh ::: param1 param2

script.sh:

host=`uname -n`
param=$1
logfile=/shared/loc/log-$host-$param
for i in `seq 1 5`; do
    touch ${logfile}_$i
    sleep 2
done

I'm trying to achieve a run on 4 processes in parallel:

When looking at the output as it accumulates it appears that what really happens is this:

-are being run in in parallel. When they finish, the other two are being run in parallel.

How can I make all four of them run in parallel at the same time?

Thanks,

Upvotes: 4

Views: 709

Answers (1)

Ole Tange
Ole Tange

Reputation: 33685

You are hitting a design decision: What does -j mean when you run --onall? The decision is that -j is the number of hosts to run on simultaneously (in your case 2). This was done so that it would be easy to run commands serially on a number of hosts in parallel.

What you can do, is wrap your parallel command with another parallel command:

parallel parallel --onall -S ${RH32},${RH64} --argsep // /shared/loc/script.sh // ::: param1 param2

This will spawn parallel for each argument and the inner parallel will spawn for each server.

Another solution is to write the ssh command yourself:

parallel ssh {1} /shared/loc/script.sh {2} ::: ${RH32} ${RH64} ::: param1 param2

Upvotes: 5

Related Questions