CODEWITHSUNDEEP
pythonamazon-web-servicesamazon-dynamodbaws-cloudformation
Matt
Matt

Reputation: 1007

"Truncating" a DynamoDB Table Created via CloudFormation (SAM)

I'd like to "truncate" (delete all items) in a DynamoDB table. I know that the most efficient way to do this would be to delete the table and re-create it (name, indexes, etc.). However, the table is part of a SAM-CloudFormation deployment. The table (by name) is also referenced within other parts of the application.

If I deleted and re-created it, I could use the same name it had previously; however, I think this would cause problems because (1) the deletion isn't immediate and (2) the ARN would change and that could have implications on the CloudFormation stack.

It seems that there should be a better solution than the brute-force approach: iterate through all items, deleting them one at a time (with some optimization via the batch_writer).

I've looked at some other solutions here, but they don't address the "part of a CloudFormation stack" part of my question.

I even provided a brute-force solution myself to another's question on this topic.

Here is the brute force approach

import boto3

table = boto3.resource('dynamodb').Table('my-table-name')
scan = None

with table.batch_writer() as batch:
    count = 0
    while scan is None or 'LastEvaluatedKey' in scan:
        if scan is not None and 'LastEvaluatedKey' in scan:
            scan = table.scan(
                ProjectionExpression='id',
                ExclusiveStartKey=scan['LastEvaluatedKey'],
            )
        else:
            scan = table.scan(ProjectionExpression='id')

        for item in scan['Items']:
            if count % 5000 == 0:
                print(count)
            batch.delete_item(Key={'id': item['id']})
            count = count + 1

The desired final state is a DynamoDB table (that was previously full of items) with the same name, no items, and still able to be destroyed as part of a CloudFormation delete operation.

Upvotes: 1

Views: 2282

Answers (1)

Dunedan
Dunedan

Reputation: 8455

No matter if you created the table as AWS::Serverless::SimpleTable or AWS::DynamoDB::Table there is no out-of-the-box solution to empty it using CloudFormation while keeping its name.

As a general best practice you shouldn't name DynamoDB tables created by CloudFormation, but let CloudFormation assign a name for the resource. If that would have been the case in your setup you could simply do a change to the resource which requires "replacement" of the resource, like temporary adding a Local Secondary Index, which would recreate the resource and would work with resources depending on it.

That said, in your situation the best approach is probably be to wrap your brute force approach in a CloudFormation custom resource and include that in your CloudFormation stack. With that you can truncate the table once or, depending on the implementation of your custom resource, whenever you want.

Keep in mind that deleting all items from a DynamoDB table might take quite long, so using a Lambda-backed custom resource might run into the limit of Lambda function runtime, depending on the number of items in the table. It might also become quite costly if the table contains a lot of items.

Upvotes: 3

Related Questions