CODEWITHSUNDEEP
amazon-web-servicesautoscaling
EmptyArsenal
EmptyArsenal

Reputation: 7464

Force terminate AutoScaling instances stuck in Pending:Wait

I have an AutoScaling group where I'm messing around with LifecycleHooks on instance start, some controlled by me and some controlled by other AWS services. I've been adding/removing LifecycleHooks and changing the size of the ASG, and instances are getting stuck in Pending:Wait. I assume it's because it's waiting for a signal from a LifecycleHook, but I can't get it to budge.

I tried a number of things:

Terminate instances in EC2 console

I terminated the instances, but ASG waits for the LifecycleHook heartbeat to timeout before actually terminating the instance from the ASG, which is probably an hour.

Complete LifecycleHooks manually

I tried to complete the LifecycleHooks manually, illustrated by the following pseudocode:

describeAutoScalingGroups -> asg {
    instances = getPending(asg.instances)
    describeLifecycleHooks -> lifecycleHooks {
        lifecycleHooks.each {
            instances.each {
                completeLifecycleAction(instance, hook)
            }
        }
    }
}

This doesn't do the trick. I'm guessing the LifecycleHook it's waiting on was deleted from the ASG, so there's no way to manually complete the LifecycleHook.

What next?

Obviously, I should be more careful about deleting these resources in the right order and all that and decreasing the heartbeat will help as well, but how can I force terminate instances from an ASG no matter what it's waiting for?

Upvotes: 12

Views: 8783

Answers (3)

GwenM
GwenM

Reputation: 1335

In case someone runs out the same problem, it depends of what your lifecycle hook is calling.

If it calls CodeDeploy then you can go to CodeDeploy and stop any running deployment that could be hanging or taking too much time. You can check which event script is pending.

Upvotes: 0

yurez
yurez

Reputation: 3222

After terminating EC2 instance in Console, manually completing LifecycleHook worked for me:

aws autoscaling complete-lifecycle-action --lifecycle-hook-name $HOOK --auto-scaling-group-name $ASG --lifecycle-action-result ABANDON --instance-id $ID

Upvotes: 18

Vivek Thomas
Vivek Thomas

Reputation: 441

Ran into this problem today, and I was able force delete using the AWS CLI.

aws autoscaling delete-auto-scaling-group --auto-scaling-group-name <asg name> --force-delete

However please note that for this to work, you must use the --force-delete argument the first time itself, if the ASG is already in a pending delete state, this will not work.

Upvotes: -3

Related Questions