CODEWITHSUNDEEP
bashgnu-parallel
Greg
Greg

Reputation: 47

run a bash for loop in parallel

I've got this script that does a credential lookup for each host, in an on-premise vault system, and then runs an ansible-playbook for it.

#!/bin/bash

for host in `cat ~/.ansible/hosts`
  do
    SECRET=`/opt/vault/bin/get-admin-credential --tag=$host`
    HOST=`echo $SECRET | cut -d ';' -f1`
    LOGIN=`echo $SECRET | cut -d ';' -f2`
    DOMAIN=`echo $SECRET | cut -d ';' -f3`
    PWD=`echo $SECRET | cut -d ';' -f4`

    if [ -z "$DOMAIN" ]; then
      ansible-playbook -i ~/.ansible/hosts ~/.ansible/windows.yml -e "ansible_host=$HOST ansible_user=$LOGIN ansible_password=$PWD" --limit $host
    else
      ansible-playbook -i ~/.ansible/hosts ~/.ansible/windows.yml -e "ansible_host=$HOST ansible_user=$LOGIN@$DOMAIN ansible_password=$PWD" --limit $host
    fi
  done

This loops over each host sequentially, I've tried stuff with GNU parallel but haven't been able to do what I want, running the for loop with 5 in parallel.

Anyone point me in the right direction?

Upvotes: 2

Views: 330

Answers (2)

chepner
chepner

Reputation: 531165

You simply need to run ansible-playbook in the background using the & command terminator. Note, though, that the entire loop can be simplified and improved.

run_playbook () {
  ansible-playbook -i ~/.ansible/hosts \
                   -e "ansible_host=$2 ansible_login=$3 ansible_password=$4" \
                   ~/.ansible/windows.yml --limit "$1"
}

while IFS= read -r host; do
    secret=$(/opt/vault/bin/get-admin-credential --tag="$host")
    IFS=";" read -r shost slogin sdomain spasswd _ <<< "$secret"
    if [[ -n $sdomain ]]; then
      login="$slogin@$sdomain"
    fi
    run_playbook "$host" "$shost" "$login" "$password" &
done < ~/.ansible/hosts

Upvotes: 1

Mark Setchell
Mark Setchell

Reputation: 207465

I don't have any "ansibles" or "vaults", so this is completely untested but may get you close:

doit(){
   host="$1"

   SECRET=$(/opt/vault/bin/get-admin-credential --tag=$host)
   HOST=$(echo $SECRET | cut -d ';' -f1)
   LOGIN=$(echo $SECRET | cut -d ';' -f2)
   DOMAIN=$(echo $SECRET | cut -d ';' -f3)
   PWD=$(echo $SECRET | cut -d ';' -f4)

   if [ -z "$DOMAIN" ]; then
      ansible-playbook -i ~/.ansible/hosts ~/.ansible/windows.yml -e "ansible_host=$HOST ansible_user=$LOGIN ansible_password=$PWD" --limit $host
   else
      ansible-playbook -i ~/.ansible/hosts ~/.ansible/windows.yml -e "ansible_host=$HOST ansible_user=$LOGIN@$DOMAIN ansible_password=$PWD" --limit $host
   fi
}

# Export doit function to subshells created by GNU Parallel
export -f doit

parallel -a ~/.ansible/hosts doit

Stylistically, there are maybe a few improvements. Firstly, shell variables consisting of upper case letters are reserved, so you shouldn't maybe use HOST, DOMAIN etc. Also, you can probably simplify all that unsightly cutting and echoing to extract the variables from the SECRET by using an IFS=';' and a read like this:

SECRET=$(/opt/vault/bin/get-admin-credential --tag=$host)
IFS=';' read host login domain pwd <<< "$SECRET"

So, my best and final answer is:

doit(){
   host="$1"

   secret=$(/opt/vault/bin/get-admin-credential --tag=$host)
   IFS=';' read host login domain pwd <<< "$secret"

   if [ -z "$domain" ]; then
      ansible-playbook -i ~/.ansible/hosts ~/.ansible/windows.yml -e "ansible_host=$host ansible_user=$login ansible_password=$pwd" --limit $host
   else
      ansible-playbook -i ~/.ansible/hosts ~/.ansible/windows.yml -e "ansible_host=$host ansible_user=$login@$domain ansible_password=$pwd" --limit $host
   fi
}

# Export doit function to subshells created by GNU Parallel
export -f doit

parallel -a ~/.ansible/hosts doit

Upvotes: 2

Related Questions