CODEWITHSUNDEEP
amazon-web-servicesaws-cloudformationaws-cdkaws-codepipeline
BK1345
BK1345

Reputation: 21

How to fix "The Following Resources Failed to Update" in Cloudformation?

I have a multi-account pipeline using AWS CodePipeline that is failing in the prod account. The pipeline will work in dev and test but fails in prod and the Cloudformation stack gives the error: "The Following Resources Failed to Update:" and lists several Lambda functions. Anyone know how to fix this? I've checked the permissions and compared them to the other accounts and they seem to match. From my understanding the resources are under the control of CF and should be able to be changed via CDK? The CDK definitions of the Lambda functions seem to match their current configurations in AWS, So I am having trouble understanding why it no longer works.

The resources also were "drifted" and to fix that I deleted the resource from the stack and made sure to include the "DeletionPolicy: "Retain" before deleting. I of course Imported those resources but I am wondering if that could be the source of the issue?

Really racking my brain with this one

Upvotes: 2

Views: 6400

Answers (1)

Farid Nouri Neshat
Farid Nouri Neshat

Reputation: 30430

For any Cloudformation failures, the first place to look at, is the "events" tab of the stack and then find what happened. There's a very nice "Detect root cause" button that normally points you to the first failure event. Most of the time the message there should be informative enough. Occasionally Cloudformation message isn't informative enough for you to know what to do.

In the case you could check what Cloudformation have been doing by checking Cloudtrail. You can then potentially see the input and the error of the failed action.

Basically every action Cloudformation does is an AWS API call behind the scenes, usually in the same region unless something is cross-region or are global services that are logged in us-east-1.

By default when you look at the Cloudtrail, in the events page it shows that recent non-readonly actions. It could take a few minutes for Cloudtrail to show all recent actions. In the events page, you can change the columns of the table to show the error code so you can quickly find which events failed. You might have to filter by user name, service of the resource or event name(the failed action name) to find the offending event/API action. Once you find the action that failed action you can then look at the input to see if it matches your expectation based on the template and potentially the error message there could have more information.

Also I suggest checking Cloudformation troubleshooting guide and also the troubleshooting guide of the service that failed to help you with further debugging.

Upvotes: 0

Related Questions