CODEWITHSUNDEEP
amazon-web-servicesaws-lambdaaws-cloudformation
Darkin Rall
Darkin Rall

Reputation: 459

AWS CloudFormation stack stuck in the state UPDATE_ROLLBACK_IN_PROGRESS

I wanted to update my stack. The stack failed with error Function not found: arn:aws:lambda.... And stack in status UPDATE_ROLLBACK_IN_PROGRESS more than 5 hours. How do I stop this process?

Upvotes: 11

Views: 37307

Answers (3)

joshuakcockrell
joshuakcockrell

Reputation: 6113

If your CloudFormation uses ECS:

For me, this happened because I had a rule for ECR images to be deleted after 1 day of inactivity. So when a CloudFormation update failed, ECS tried to rollback to an image that wasn't there anymore.

I manually deleted the ECS service (via AWS web UI) and waited ~1 hour for it to timeout. Then I could delete the CloudFormation stack.

This is why you test deploys in staging first!

Upvotes: 1

Debrup Majumdar
Debrup Majumdar

Reputation: 114

In our case, we have mistakenly passed wrong image name to cloudformation template. After realising the mistake, we tried to stop the stack update, which made the stack stuck for forever in UPDATE_ROLLBACK_IN_PROGRESS status. SO during ECS service creation it got stuck. Solution:

  1. in Stack event check in which step is in progress. (our case ECS service update)
  2. Go to ECS service.
  3. Click on Update service.
  4. Choose older task definitions.
  5. And Update.

Your Task definition is reset to previous version. And roll back will complete successfully.

Upvotes: 5

Marcin
Marcin

Reputation: 238209

If you deleted the function outside of CloudFormation, then you can manually create a new function of the same name. This sometimes helps.

You can also wait till the rollback timeouts. And it usually does after a while, but the time varies.

Another reason why it gets stuck in this state could be due to nested stacks:

In this case a recommended option is indeed to contact support:

To fix the stack, contact AWS customer support.

Recent AWS blog post also describes the issue and possible solutions:

Regarding the time to wait, the timeout varies:

In most situations, you must wait for your AWS CloudFormation stack to time out. The timeout length varies, and is based on the individual resource stabilization requirements that AWS CloudFormation waits for to reach the desired state.

Upvotes: 4

Related Questions