CODEWITHSUNDEEP
bashshellamazon-web-services
max scalf
max scalf

Reputation: 329

for loop with batch of 5 in the loop

I am writing a shell script that copies our AWS EBS Snapshot from one region to another region. It's a simple thing that i am doing, which is running a for loop with below command from a list of ID that are stored in a file(snap_id).

for SID in `cat snap_id`
do
    aws --region us-east-1 ec2 copy-snapshot --source-region us-west-2 --source-snapshot-id $SID --description "copy to us-west"
done

Here is a sample snap_id file(100's of line of snap-id)

snap-5ddffd59
snap-9c054999
snap-d94496d9
snap-er2df342
snap-234as234
snap-as234asd
......
......
......

Now the problem i am running into is, I can only run 5 of those at any given time as AWS limit how many snapshot you can copy to another region concurrently.

So basically i have to create a queue of 5, and I have to wait until those 5 are done before i send the batch of next 5. I can run something like below command to get a status of current copy(which should be "completed", we would get 5 completed if things are complete)

aws ec2 describe-snapshots --snapshot-ids (LIST OF CURRENT 5 BATCH) --query Snapshots[].State --output text

so any idea on how to build a queue like this using bash ?

UPDATE 2: After suggestion from @jyvet

I changed my waitbatch to below, In that way i have a timeout of 1800 seconds and i look for all of the 5 (in that batch) to have a complete status and increment the counter to 5, if the count is 5 (5 out of 5 in the current batch) i break out... Suggestions are welcomed ...

waitbatch() {
count=0

if [ ${ibatch} -gt 0 ]; then
  END=$((SECONDS+1800))
  while [ $SECONDS -lt $END ]; do
    while [ "$count" -le 5 ]; do
    # look for errors ??? and put retries/timeout
      aws ec2 describe-snapshots --snapshot-ids ${list} --query Snapshots[].State --output text > completed_list
      count=0
      for i in `cat completed_list`
      do
        if [ "$i" = "completed" ]
        then
          count=$((count+1))
        fi
      done
      if [ "$count" = "5" ]
      then
        echo count is good, ready to move to next batch
        break
      fi
    echo "sleeping 5 seconds"
    sleep 5
    done
    if [ "$count" = "5" ]
    then
      break
    fi
  done
fi
}

Upvotes: 0

Views: 1320

Answers (1)

jyvet
jyvet

Reputation: 2201

What's about something like:

#!/bin/bash

BATCH=5      # nb of snap ids in a batch 
TIMEOUT=10   # seconds to wait for completion of a batch

ibatch=0

waitbatch() {
    retries=0

    if [ ${ibatch} -gt 0 ]; then
        while [ `aws ec2 describe-snapshots --snapshot-ids (${list}) --query Snapshots[].State --output text | grep -o 'completed' | wc -w` -lt ${BATCH} ]; do
            sleep 1

            retries=$(( ${retries} + 1))

            # Give up if TIMEOUT
            if [ ${retries} -eq ${TIMEOUT} ]; then
                >&2 echo "timeout for ${list}"
                break
            fi
        done
    fi
}


# Loop over all snap ids
for SID in `cat snap_id`
do
    aws --region us-east-1 ec2 copy-snapshot --source-region us-west-2 --source-snapshot-id ${SID} --description "copy to us-west"
    list="${list} ${SID}"

    ibatch=$(( ${ibatch} + 1 ))

    # Check if BATCH is reached
    if [ ${ibatch} -eq ${BATCH} ]; then

        # Wait for completion
        waitbatch

        ibatch=0
        list=""
    fi
done


# Wait for completion
waitbatch

Upvotes: 2

Related Questions