CODEWITHSUNDEEP
amazon-web-servicesamazon-ec2aws-cloudformation
chris
chris

Reputation: 37470

Is there a way to get a volume unmounted when I call CloudFormation delete-stack?

I am using CloudFormation to create my environment. Part of the stack includes creating a volume from a snapshot, associating it with an EC2 instance, and then mounting it.

"Resources" : {
    "EBSData" : {
        "Type" : "AWS::EC2::Volume",
     ...snip...
    },

    "MountPoint" : {
        "Type" : "AWS::EC2::VolumeAttachment",
     ...snip...
    },

    "myTestInstance" : {
        "Type" : "AWS::EC2::Instance",
     ...snip...
    }
 },

When I try to call delete-stack, it fails because the volume is still mounted:

  "StackStatusReason": "The following resource(s) failed to delete: [EBSData, MountPoint].", 
  "CreationTime": "2013-12-03T13:40:58.646Z", 
  "StackName": "myTestStack", 
  "StackStatus": "DELETE_FAILED", 
  "DisableRollback": false

Calling delete-stack a 2nd time succeeds, because the instance has been destroyed already.

Are there any hooks into the running instance that get invoked by cloudformation delete-stack where I could un-mount the volume? Any other way to do this?

Upvotes: 10

Views: 3771

Answers (3)

magnetik
magnetik

Reputation: 4431

What CloudFormation does not do, it cannot undo: your volume is mounted by you (with CloudInit via UserData or else), and CloudFormation has not idea on how to undo this. Thus the stack deletion is stopped because the device stays mouted, and the volume is in "Busy state".

The recommended way is to use CloudFormation CustomRessource to declare the mount point:

"ExampleVolumeMount" : {
    "Type" : "Custom::VolumeMount",
    "Version" : "1.0",
    "DependsOn" : ["VolumeAttachment", "ExampleWaitCondition"],
    "Properties" : {
        "ServiceToken" : { "Fn::GetAtt" : ["CustomResourcePipeline", "Outputs.CustomResourceTopicARN"] },
        "Device" : "/dev/xvdh",
        "MountPoint" : "/mnt/disk",
        "FsType" : "ext3",
        "Format" : "true"
    }
},

It uses a ServiceToken created by a sub template: https://s3.amazonaws.com/cloudformation-examples/cr-backend-substack-template.template which contain a SQS queue and a SNS topic.

You can check the complete example and scripts on AWSlabs github account.

Upvotes: 1

Dermah
Dermah

Reputation: 61

To get this working, I had to change the way I attached the volume. Instead of using a AWS::EC2::VolumeAttachment, you can specify EBS volumes to attach to an instance within the properties of that instance:

"Resources" : {
    "EBSData" : {
        "Type" : "AWS::EC2::Volume",
     ...snip...
    },

    "myTestInstance" : {
        "Type" : "AWS::EC2::Instance",
        "Properties": {
            "Volumes": [ { "VolumeId": { "Ref": "EBSData" }, "Device": "<device mount point>" }]
        ...snip...
        },
     ...snip...
    }
 },

Attaching the volume this way appears to make CloudFormation delete the instance and volume in the correct order. That is, the instance is shut down before the volume is deleted.

Be sure that EBSData does not use any references to myTestInstance or you'll get a circular dependency.

Upvotes: 6

David Sulpy
David Sulpy

Reputation: 2347

I found the answer to my problem here with the "DependsOn" resource attribute.

Essentially, this was able to solve my problem (described in a comment to your question):

"Resources" : {
    "EBSData" : {
        "Type" : "AWS::EC2::Volume",
        "DependsOn": "myTestInstance",
     ...snip...
    },
    "MountPoint" : {
        "Type" : "AWS::EC2::VolumeAttachment",
     ...snip...
    },

    "myTestInstance" : {
        "Type" : "AWS::EC2::Instance",
     ...snip...
    }
 },

Note the addition of "DependsOn": "myTestInstance" in the Volume resource. Since the VolumeAttachment resource has a built in "DependsOn" for the Volume, all should be good.

Additional Note: This hasn't been tested, but will update once it's tested.

Upvotes: 3

Related Questions