CODEWITHSUNDEEP
bashgnu-parallel
Pierre-Antoine DOUCHET
Pierre-Antoine DOUCHET

Reputation: 57

GNU Parallel - Alternate argument

I want to be able to alternate an argument for each execution of my parallel command.

I have a script to process images and I use GNU Parallel to launch several process at the same time. I use GPU core in this script but only one of my 2 GPU core is used.

I want to be able to use 2 different configuration file (one for GPU 1 et one for GPU 2) but I don't know how to do that.

gsutil ls -d $ee_repo | parallel -j 10 -k "task {} $PWD $year $config" With $config changing from gpu_1 to gpu_2 every other execution

Is this possible or not ?

Thanks 😉

EDIT:

Using the job number of the parallel command ({#}) as an argument, then get 0 or 1 by using the modulo of this job number:

gsutil ls -d $ee_repo | parallel -j 10 -k "task {} $PWD $year {#}"

In the task function:

task(){
...
jobnumber=$4    
gpu=$(expr $jobnumber% 2)
...
}

Upvotes: 2

Views: 163

Answers (2)

Ole Tange
Ole Tange

Reputation: 33685

Given your description you probably do not want GNU Parallel to alternate. Imagine that all odd jobs take 1 second and all even jobs take 2 seconds. Then you end up with jobs running on the same GPU while none are running on the other.

What you instead want is to have a job starting on GPU2 as soon as a job finished on GPU2. {%} (aka. job slot) is made for this:

parallel -j2 'sleep {}; echo {}-GPU{%} seq {#}' ::: 1 2 1 2 1 2 1 2 1 2 1 2

Upvotes: 1

Mark Setchell
Mark Setchell

Reputation: 207425

Something like this should work:

seq 14 | parallel -k echo task {} {= '$_=seq() % 2 ? "gpu1" : "gpu2"' =}

Output

task 1 gpu1
task 2 gpu2
task 3 gpu1
task 4 gpu2
task 5 gpu1
task 6 gpu2
task 7 gpu1
task 8 gpu2
task 9 gpu1
task 10 gpu2
task 11 gpu1
task 12 gpu2
task 13 gpu1
task 14 gpu2

Upvotes: 0

Related Questions