CODEWITHSUNDEEP
amazon-web-servicesamazon-ec2aws-cloudformation
Rome_Leader
Rome_Leader

Reputation: 2700

Can't edit CloudFormation stack after failed update?

My CloudFormation stack seems to have exploded. I tried to make a quick change to resize the EC2 instances and the update failed with the error:

Invalid value 'm4.large' for instanceType. LaunchPlan instance type does not match attribute value m4.xlarge

The rollback also failed with error:

The following resource(s) failed to update: [TC2, TC1, TC3].

My only option when right-clicking the stack is to delete, and trying to continue the update by force through the actions menu yields:

This instance 'i-e5e8b063' is not in a state from which it can be stopped.

Looking at the EC2 Instance console, I can see that this message is at least true: All the instances are listed as terminated.

All the suggestions here about what to do about a failed update seem to predicated on me actually being able to modify the stack to fix the error first, which I'm unable to do: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/troubleshooting.html#troubleshooting-errors-update-rollback-failed

Do I any recourse besides blowing away the stack and designing a new one from my backup? I'm at a total loss.

Upvotes: 5

Views: 8586

Answers (1)

Sam King
Sam King

Reputation: 2188

CloudFormation tries to be declarative and tries to be able to move from wherever you start to the state in your template. Sometimes, it gets into an inconsistent state (when given a bad template or when manual changes are made outside of CloudFormation). When this happens to me, CloudFormation usually rolls back properly, but when it doesn't, I have only ever succeeded in just deleting the stack and starting from scratch.

As a consequence of that, two things are important in CloudFormation:

  1. Using multiple tiers. Eg, a FooDev and a FooProd CloudFormation stack so that you can test things out (especially upgrades and such) in Dev rather than breaking Prod.
  2. Using multiple stacks. Eg, your database stack should be long lived and stateful, so it might get a stack of its own and you should be careful with them, whereas your EC2 instances should be short lived and stateless, so it's much more okay if you mess up a deploy of one of those.

Those best practices as well as others are detailed at https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/best-practices.html

Upvotes: 6

Related Questions