CODEWITHSUNDEEP
amazon-web-servicesamazon-ec2aws-lambdadistributed-system
Jeff Bezos
Jeff Bezos

Reputation: 2253

Launch AWS instance on upload

I want to host a website on a small AWS instance (e.g. t2.micro) that launches a larger instance (e.g. m5.2xlarge) whenever a user uploads data to be processed. I also want the larger instance to shut down when the processing is finished. Is there an AWS service or some other workaround that can do this? The larger instance would only be triggered every few days.

Upvotes: 0

Views: 121

Answers (2)

Jeff Bezos
Jeff Bezos

Reputation: 2253

You can also point the instance to the correct S3 bucket during launch. Just provide a script to the UserData argument:

# Point to bucket.
parent = 'my-awesome-s3-bucket'
startup_script = '''#!/bin/bash
bucket=<PARENT_NAME>
aws s3 cp s3://$bucket /home/ubuntu --recursive
'''
startup_script = startup_script.replace('<PARENT_NAME>', parent)

# Launch instance.
try:    
    response = ec2.run_instances(
        InstanceType='m4.2xlarge',
        KeyName='aws3',
        SecurityGroups=['launch-wizard-12'],
        MinCount=1,
        MaxCount=1,
        UserData=startup_script
    )

Upvotes: 0

John Rotenstein
John Rotenstein

Reputation: 269490

The easiest way to stop an instance when it has finished a task is to issue a shutdown command to the Operating System. The default behaviour will be to Stop the EC2 instance.

The easiest way to start the instance is to call the StartInstances() command, either via command-line or via an SDK. The tricky part is telling the instance what data should be processed. Also, consider what should happen if another data file is uploaded while the first file is being processed -- should it shutdown, or should it work on the new file?

I would recommend:

  • If the file is uploaded to S3, then use an AWS Lambda function to start the bigger instance (not requiring the smaller instance)
  • If the file is uploaded to the smaller instance, then that instance should:
    • Push a message to an Amazon SQS queue containing the data to be processed (or copy the file to S3 and then have the message point to the object in S3)
    • Call StartInstances() to start the bigger instance
  • The bigger instance should then enter a loop where it will:
    • Retrieve a message from the Amazon SQS queue
    • Process the data
  • If the SQS queue is empty, then shutdown the instance via the operating system

See: Auto-Stop EC2 instances when they finish a task - DEV

Upvotes: 1

Related Questions