Unable to add GSI to DynamoDB table using CloudFormation

Question

I have an existing DynamoDB table that is defined as part of a CloudFormation stack. According the the CFN AWS::DynamoDB::Table documentation the GlobalSecondaryIndexes attribute does not require replacement. It even goes into details with the following caveats.

You can delete or add one global secondary index without interruption.

As well as the following...

If you update a table to include a new global secondary index, AWS CloudFormation initiates the index creation and then proceeds with the stack update. AWS CloudFormation doesn't wait for the index to complete creation because the backfilling phase can take a long time, depending on the size of the table.

However, in practice when I attempt to perform an update I get the following error message:

CloudFormation cannot update a stack when a custom-named resource requires replacing. Rename mytablename and update the stack again.

Since I'm adding a GSI that uses a new attribute I'm forced to modify AttributeDefinitions which says it does require replacement. However, even when I try to add a GSI with only existing attributes defined in the AttributeDefinitions I still get the same error message.

Here is the snippet from my original CFN definition for my table:

{
  "myTable": {
    "Type": "AWS::DynamoDB::Table",
    "Properties": {
      "TableName": "mytablename",
      "AttributeDefinitions": [
        {
          "AttributeName": "entryId",
          "AttributeType": "S"
        },
        {
          "AttributeName": "entryName",
          "AttributeType": "S"
        },
        {
          "AttributeName": "appId",
          "AttributeType": "S"
        }
      ],
      "KeySchema": [
        {
          "KeyType": "HASH",
          "AttributeName": "entryId"
        },
        {
          "KeyType": "RANGE",
          "AttributeName": "entryName"
        }
      ],
      "ProvisionedThroughput": {
        "ReadCapacityUnits": {
          "Ref": "readThroughput"
        },
        "WriteCapacityUnits": {
          "Ref": "writeThroughput"
        }
      },
      "GlobalSecondaryIndexes": [
        {
            "IndexName": "appId-index",
          "KeySchema": [
            {
              "KeyType": "HASH",
              "AttributeName": "appId"
            }
          ],
          "Projection": {
            "ProjectionType": "KEYS_ONLY"
          },
          "ProvisionedThroughput": {
            "ReadCapacityUnits": {
              "Ref": "readThroughput"
            },
            "WriteCapacityUnits": {
              "Ref": "writeThroughput"
            }
          }
        }
      ]
    }
  }
}

Here is what I want to update it to:

{
  "myTable": {
    "Type": "AWS::DynamoDB::Table",
    "Properties": {
      "TableName": "mytablename",
      "AttributeDefinitions": [
        {
          "AttributeName": "entryId",
          "AttributeType": "S"
        },
        {
          "AttributeName": "entryName",
          "AttributeType": "S"
        },
        {
          "AttributeName": "appId",
          "AttributeType": "S"
        },
        {
          "AttributeName": "userId",
          "AttributeType": "S"
        }
      ],
      "KeySchema": [
        {
          "KeyType": "HASH",
          "AttributeName": "entryId"
        },
        {
          "KeyType": "RANGE",
          "AttributeName": "entryName"
        }
      ],
      "ProvisionedThroughput": {
        "ReadCapacityUnits": {
          "Ref": "readThroughput"
        },
        "WriteCapacityUnits": {
          "Ref": "writeThroughput"
        }
      },
      "GlobalSecondaryIndexes": [
        {
            "IndexName": "appId-index",
          "KeySchema": [
            {
              "KeyType": "HASH",
              "AttributeName": "appId"
            }
          ],
          "Projection": {
            "ProjectionType": "KEYS_ONLY"
          },
          "ProvisionedThroughput": {
            "ReadCapacityUnits": {
              "Ref": "readThroughput"
            },
            "WriteCapacityUnits": {
              "Ref": "writeThroughput"
            }
          }
        },
        {
          "IndexName": "userId-index",
          "KeySchema": [
            {
              "KeyType": "HASH",
              "AttributeName": "userId"
            }
          ],
          "Projection": {
            "ProjectionType": "KEYS_ONLY"
          },
          "ProvisionedThroughput": {
            "ReadCapacityUnits": {
              "Ref": "readThroughput"
            },
            "WriteCapacityUnits": {
              "Ref": "writeThroughput"
            }
          }
        }
      ]
    }
  }
}

However, like I mentioned before even if I do not define userId in the AttributeDefinitions and use an existing attribute in a new GSI definition it does not work and fails with the same error message.

Answer 1

How the issue happened here? For me, delete the GSI manually in dynamoDB console, then add GSI by cloudformation, update-stack got this error.

Solution: remove the GSI in cloudformation, execute update-stack, then add back the GSI, execute update-stack again, works fine.

Guess cloudformation has its own cache, could not tell the change you've done manually in console.

Answer 2

My scenario was that I wanted to update a GSI by chnaging its range key. - First you have to delete the GSI that you're updating, also remember to remove any AttributeDefinition that might not be needed anymore due to the removal of the GSI i.e. the index name etc. Upload the template via CloudFormation to apply the changes. - Then add the needed Attributes and the 'updated' GSI to the template.

Answer 3

Backup all the data from the DynamoDB and after that, if you are using serverless, perform either of the command below:

individual remove:

node ./node_modules/serverless/bin/serverless remove

globally remove:

serverless remove

and deploy it again by running:

node ./node_modules/serverless/bin/serverless deploy -v

or

serverless deploy

Answer 4

AWS support's response to me FWIW:

Workaround A

1. Export data from the table to s3
2. Update stack with new tablename (tablename2) with gsi added
   • Note this losses all current entries, so definitely backup to s3 first!
3. Update stack again, back to using tablename1 in the dynamodb table
4. Import data from s3 This can be eased by using data pipelines, see
   • http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBPipeline.html

Advantage is, app code can keep using fixed names. But updating the stack twice and exporting/importing data will take some work to automate in custom scripts.

Workaround B

1. Backup data
2. Let CloudFormation name the table
3. Use the AWS SDK to retrieve the table name by getting the name through describing the stack resource by logical-id, and fetching from the output the tablename there.

While I think this avoids extra stack updates (still think exporting/importing data will be required), the disadvantage is a network call in code to fetch the table name. See * http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/CloudFormation.html#describeStackResource-property

Again, this is a known issue support is pushing the service team on, as we know it is a quite common use case and point of pain. Please try a workaround on a test environment before testing on production.

Answer 5

I had the same error today and got an answer from Amazon tech support. The problem is that you supplied a TableName field. CloudFormation wants to be in charge of naming your tables for you. Apparently, when you supply your own name for them, this is the error you get on a update that replaces the table (not sure why it needs to replace, but that's what the doc says)

For me, this makes CloudFormation utterly useless for maintaining my DynamoDB tables. I'd have to build in configuration so that my code could dynamically tell what the random table name was that CloudFormation generated for me.