CODEWITHSUNDEEP
amazon-web-servicesdockeraws-ebs
Atakan E.
Atakan E.

Reputation: 215

Is it possible to trigger instance termination on AWS if an EBS volume gets detached?

I have a stateful cluster deployed on AWS in which instances attach to an already existing EBS volume on startup and this volume would later be mounted to the Docker container running on the instance. If I forcefully detach this volume, the instance as well as the Docker container continue to be functional. To attach to the same volume, the instance has to be terminated and the new instance launched by the autoscaling group would attach to the detached volume through the userdata script.

Is there a way to automatically detect volume detachments and trigger an attachment? Or is it possible to automatically kill the instance if its EBS volume is forcefully detached?

Upvotes: 0

Views: 164

Answers (1)

MisterSmith
MisterSmith

Reputation: 3624

I dont know of any automatic way to achieve this out-of-the-box. Best i can offer are a few ideas to investigate.

  1. Run a cron script on your docker hosts that checks if the path is still accessible every X minutes. If path is not accessible, and if instances are set to terminate on shutdown just call shutdown -h to kill it. Or use the AWS CLI from your docker hosts to request the current instance is killed. A script can get the current instances InstanceId at runtime from the instance MetaData via curl, and you will need an IAM Policy and assign it to an IAM role for the instance to gain permission to terminate an instance.
  2. Basically same thing, but do it from another server, or a Lambda function on a schedule that queries the API to get a list of instances / volumes(based on tag etc), and then checks the attachment status and terminate an instance if necessary.
  3. Depending on your use case, you could maybe use cloudwatch to monitor the EBS metrics for the volume. could you detect a failure based on this for your use case then execute a lambda to actually inspect the instance and terminate it?

Upvotes: 1

Related Questions