CODEWITHSUNDEEP
amazon-web-servicesamazon-ec2
kgautron
kgautron

Reputation: 8263

Replacement of a spot instance about to be terminated by an on-demand instance

According to AWS documentation, the termination of a SPOT instance is notified 2 minutes in advance. I would like to make sure my service keeps running, by replacing a SPOT instance that is notified to be deleted, by an on-demand instance.

Are there existing ways (either built-in AWS configuration or external tools) to automatically replace an EC2 SPOT instance by an on-demand instance when the SPOT instance is marked for termination?

Upvotes: 2

Views: 1680

Answers (3)

SenseDeep
SenseDeep

Reputation: 3086

Full disclosure: I work for SenseDeep that provides the PowerDown service that does automatic replacement of Spot instances.

PowerDown will receive the 2 minute warning and will proactively spin up an On-Demand instance in an AutoScale group to maintain capacity in AutoScale groups. When the Spot market recovers, it will automatically replace the OnDemand instance with a Spot instance. You can specify the number or percentage of On-Demand and Spot instances that you want maintained.

In this way, you can use Spot instances in production as PowerDown will maintain availability for your AutoScale group.

With the latest AWS spot market pricing update (2018), spot instances are living much longer (often weeks at a time), but other authors are correct: that if you absolutely, 99.9999999% must guarantee that you have an instance -- go with reserved instances.

Upvotes: 0

E.J. Brennan
E.J. Brennan

Reputation: 46849

I don't think its possible to convert your running spot instance into an on-demand instance (if that is what you are asking), but if what you really meant is to spin-up a new ondemand instance, then generally you could do the following:

You should run a simple cron job on your instance that checks every 5 seconds for the termination notice, i.e.

#!/bin/bash
while true
    do
        if [ -z $(curl -Is http://169.254.169.254/latest/meta-data/spot/termination-time | head -1 | grep 404 | cut -d \  -f 2) ]
            then
                # Call your script to launch on-demand instance here.
                break
            else
                # Spot instance not yet marked for termination.
                sleep 5
        fi
    done

Upvotes: 5

Julio Faerman
Julio Faerman

Reputation: 13501

Although there are some tools for spot bidding and management, i don't think one would do exactly that. I'd recommend:

  • Don't do this in production/mission critical/can't outage scenarios. Having one single instance that can be terminated anytime would be dangerous (for example if this automation fails somehow)
  • Use a Spot Fleet with varied instance types to keep at desired size automatically and efficiently (1 in this case).
  • An alternative to spot fleet would be two auto scaling groups, one for spot and one on-demand (both min=0 max=1), and auto scaling policies to resize on CloudWatch alarms.

Upvotes: 2

Related Questions