CODEWITHSUNDEEP
linuxbashshellparallel-processingscripting
John Doe
John Doe

Reputation: 159

Executing a for loop with gnu parallel

I'd like to use gnu parallel to exec a function inside a bash script in a for loop and I'm not able to figure out how from the info I got online . For example:

#!/bin/bash

get_racks(){
query that outputs a list of racks 
}

get_hosts() {
query that outputs a list of hosts by rack(passed as param to the func)
}

get_gw() {
query that outputs a list of gateways
}

check_ping() {
HOSTS=(`get_hosts`)
COUNTER=0
while [ $COUNTER -lt $SIZE ] ; do
  ssh $HOSTS "ping -c 5 ${GATEWAYS[$COUNTER]} " ;
  COUNTER=$((COUNTER+1)) ;
done
}

RACKS=(`get_racks`)
HOSTS=(`get_hosts`)
GWS=(`get_gw`)
SIZE=${#GWS[@]}
COUNTER=0

for rack in ${RACKS[@]}; do
  parallel check_ping | tee output.txt 
done

I'd like to execute the for loops and what happens inside in parallel while controlling the number of concurrent jobs and output everything into a single file . The final file should have all the data the function returns and also print to stdout . Also , the check_ping has a part of sshing to a remote machine and pinging a GW, this part happens actually in parallel with a custom ssh command we have that ssh to all the hosts in the array in parallel and pings a GW .

Thanks much

Upvotes: 1

Views: 1651

Answers (3)

Ole Tange
Ole Tange

Reputation: 33748

#!/bin/bash

get_racks(){
    # query that outputs a list of racks
    echo rack01
}
export -f get_racks

get_hosts() {
    # query that outputs a list of hosts by rack(passed as param to the func)
    echo host01
    echo host02
}
export -f get_hosts

get_gw() {
    # query that outputs a list of gateways
    echo gw1
    echo gw2
}
export -f get_gw

ping_gw() {
    rack="$1"
    get_hosts "$rack" | parallel -S - --onall ping -c 5 ::: "`get_gw`"
}
export -f ping_gw
get_racks | parallel ping_gw && echo All hosts and gws are up

If you have more functions/variables/aliases then consider using env_parallel instead. That way env_parallel will do the exporting for you.

Upvotes: 1

Mark Setchell
Mark Setchell

Reputation: 208043

I think you want this:

RUN_COMMANDS() {
...
}

export -f RUN_COMMANDS

parallel -j4 RUN_COMMANDS ::: “${FILES[@]}” | tee output.txt

You will want parallel -k ... if the order is important.

Upvotes: 2

ntj
ntj

Reputation: 171

Parallel can be used in a very similar way to xargs, it even has the params of xargs implemented.

#!/bin/bash

RUN_COMMANDS(){
    ...... 
}

FILES=(file1 file2 file3)

echo ${FILES[@]} | parallel RUN_COMMANDS | tee output.txt

Upvotes: 0

Related Questions