nu everest
nu everest

Reputation: 10249

AWS EC2: How to prevent infinite loop for instance status check?

I have the following python boto3 code with a potentially infinite while-loop. Generally, after a few minutes the while-loop succeeds. However, if something fails on the AWS side the program could hang for an indefinite period.

I am sure that this is not the most appropriate way to do this.

# credentials stored in ../.aws/credentials
# region stored in ../.aws/config

# builtins
from time import sleep
# plugins
import boto3

# Assign server instance IDs.
cye_production_web_server_2 = 'i-FAKE-ID'

# Setup EC2 client
ec2 = boto3.client('ec2')

# Start the second web server.
start_response = ec2.start_instances(
    InstanceIds=[cye_production_web_server_2, ],
    DryRun=False
)

print(
    'instance id:',
    start_response['StartingInstances'][0]['InstanceId'],
    'is',
    start_response['StartingInstances'][0]['CurrentState']['Name']
)

# Wait until status is 'ok'
status = None
while status != 'ok':
    status_response = ec2.describe_instance_status(
        DryRun=False,
        InstanceIds=[cye_production_web_server_2, ],
    )
    status = status_response['InstanceStatuses'][0]['SystemStatus']['Status']
    sleep(5)    # 5 second throttle

print(status_response)
print('status is', status.capitalize())

Upvotes: 0

Views: 1752

Answers (3)

skantana
skantana

Reputation: 11

Use timeout might be a better idea

import time 

systemstatus = False
timeout = time.time() + 60*minute

while systemstatus is not True:
    status = ec2.describe_instance_status( \
                 DryRun = False,
                 InstanceIds = [instance_id]
             )

    if status['InstanceStatuses'][0]['SystemStatus']['Status'] == 'ok':
        systemstatus = True

    if time.time() > timeout:
        break
    else:
        time.sleep(10)

Upvotes: 0

Vorsprung
Vorsprung

Reputation: 34327

Implement a counter in the loop and fail after so many attempts

status = None
counter = 5
while (status != 'ok' and counter > 0):
    status_response = ec2.describe_instance_status(
        DryRun=False,
        InstanceIds=[cye_production_web_server_2, ],
    )
    status = status_response['InstanceStatuses'][0]['SystemStatus']['Status']
    sleep(5)    # 5 second throttle
    counter=counter-1

print(status_response)
print('status is', status.capitalize())

Upvotes: 3

Kevin London
Kevin London

Reputation: 4728

You could try doing it in a for loop instead with a fixed number of tries.

For example:

MAX_RETRIES = 5

# Try until status is 'ok'
for x in range(MAX_RETRIES):
    status_response = ec2.describe_instance_status(
        DryRun=False,
        InstanceIds=[cye_production_web_server_2, ],
    )
    status = status_response['InstanceStatuses'][0]['SystemStatus']['Status']
    if status != 'ok':
        sleep(5)    # 5 second throttle
    else:
        break

Upvotes: 0

Related Questions