CODEWITHSUNDEEP
linuxbashparallel-processinggnu-parallel
David
David

Reputation: 10152

Multiple (with one fixed) GNU-parallel arguments

I use GNU-parallel to call a function foo repeatedly. The function takes two arguments, a filename (a in the example below, saved in the array A) and an array of filters (B below, with b as elements of B).

The function should iterate over all combinations of A and B, however, the catch is, that I actually use parallel only to iterate over A, while providing B to each function call.

As a minimum example:

#!/bin/bash

A=("1" "2" )
B=("X" "Y")

foo() {
    a=$1 # a single element of A
    B=$2 # the whole array B
    for b in "${B[@]}"; do
        printf "a = %s; b = %s\n" $a $b
    done
    echo "-----------------"
}
export -f foo

# goal:
echo "Sequential (aka, the target) ==="
for a in "${A[@]}"; do
    foo $a $B
done

Results

Sequential (aka, the target) ===
a = 1; b = X
a = 1; b = Y
-----------------
a = 2; b = X
a = 2; b = Y
-----------------

Mind that instead of having a single call to each combination, we have only for each A a call and then within the function iterate over B.

Parallel tries:

Try 1

parallel foo ::: "${A[@]}" ::: "${B}"

Results in

a = 1; b = X
-----------------
a = 2; b = X
-----------------

(missing the second argument of B)

Try 2

parallel foo ::: "${A[@]}" ::: "${B[@]}"

Results in

a = 1; b = X
-----------------
a = 1; b = Y
-----------------
a = 2; b = X
-----------------
a = 2; b = Y
-----------------

(one call per combination, not one call per A, then iterating over B)

I have looked through the manual and SO but couldn't find a solution.

Edit

I have the feeling that it should work when I export the array B directly, but get no results as well

foo2() {
    a=$1 # a single element of A
    # B=$2 # the whole array B
    for b in "${B[@]}"; do
        printf "a = %s; b = %s\n" $a $b
    done
    echo "-----------------"
}
export -f foo2
export B

parallel foo ::: "${A[@]}"

Results in

-----------------
-----------------

(apparently empty B)

Upvotes: 2

Views: 1131

Answers (1)

Ole Tange
Ole Tange

Reputation: 33685

Your problem stems from B being an array. I have never seen an array being passed as an argument to a function before, and I am not sure it can be done.

The second issue is that while you can export both functions and variables, you cannot export an array without cheating: Exporting an array in bash script

GNU Parallel makes it easy to cheat by using env_parallel. This will export the full environment to the shell running the command: Arrays, variables, aliases, and functions. It will even do so when running commands remotely.

env_parallel has improved a lot the past year, so upgrade if the version you have causes problems.

#!/bin/bash                                                                        

A=("1" "2" )
B=("X" "Y")

foo() {
    a=$1 # a single element of A                                                   
#    B=$2 # the whole array B                                                      
    for b in "${B[@]}"; do
        printf "a = %s; b = %s\n" $a $b
    done
    echo "-----------------"
}

# If you have earlier run 'env_parallel --install'                                 
# to activate env_parallel in your shell                                           
# this should work.                                                                
env_parallel foo ::: "${A[@]}"

# If you have not, then this should work:
. `which env_parallel.bash`
env_parallel foo ::: "${A[@]}"

Upvotes: 2

Related Questions