CODEWITHSUNDEEP
amazon-dynamodb
Dimitre Novatchev
Dimitre Novatchev

Reputation: 243549

Change the schema of a DynamoDB table: what is the best/recommended way?

What is the Amazon-recommended way of changing the schema of a large table in a production DynamoDB?

Imagine a hypothetical case where we have a table Person, with primary hash key SSN. This table may contain 10 million items.

Now the news comes that due to the critical volume of identity thefts, the government of this hypothetical country has introduced another personal identification: Unique Personal Identifier, or UPI.

We have to add an UPI column and change the schema of the Person table, so that now the primary hash key is UPI. We want to support for some time both the current system, which uses SSN and the new system, which uses UPI, thus we need both these two columns to co-exist in the Person table.

What is the Amazon-recommended way to do this schema change?

Upvotes: 73

Views: 50323

Answers (4)

vinay
vinay

Reputation: 63

If the changes involve changing the partition key, you can add a new GSI (global secondary index). Moreover, you can always add new columns/attributes to DynamoDB without needing to migrate tables.

Upvotes: 3

Abhaya Chauhan
Abhaya Chauhan

Reputation: 1201

DynamoDB streams enable us to migrate tables without any downtime. I've done this to great effective, and the steps I've followed are:

  1. Create a new table (let us call this NewTable), with the desired key structure, LSIs, GSIs.
  2. Enable DynamoDB Streams on the original table
  3. Associate a Lambda to the Stream, which pushes the record into NewTable. (This Lambda should trim off the migration flag in Step 5)
  4. [Optional] Create a GSI on the original table to speed up scanning items. Ensure this GSI only has attributes: Primary Key, and Migrated (See Step 5).

  5. Scan the GSI created in the previous step (or entire table) and use the following Filter:

    FilterExpression = "attribute_not_exists(Migrated)"

Update each item in the table with a migrate flag (ie: “Migrated”: { “S”: “0” }, which sends it to the DynamoDB Streams (using UpdateItem API, to ensure no data loss occurs).

NOTE: You may want to increase write capacity units on the table during the updates.

  1. The Lambda will pick up all items, trim off the Migrated flag and push it into NewTable.
  2. Once all items have been migrated, repoint the code to the new table
  3. Remove original table, and Lambda function once happy all is good.

Following these steps should ensure you have no data loss and no downtime.

I've documented this on my blog, with code to assist: https://www.abhayachauhan.com/2018/01/dynamodb-changing-table-schema/

Upvotes: 28

Brian
Brian

Reputation: 2725

I'm using a variant of Alexander's third approach. Again, you create a new table that will be updated as the old table is updated. The difference is that you use code in the existing service to write to both tables while you're transitioning instead of using a lambda function. You may have custom persistence code that you don't want to reproduce in a temporary lambda function and it's likely that you'll have to write the service code for this new table anyway. Depending on your architecture, you may even be able to switch to the new table without downtime.

However, the nice part about using a lambda function is that any load introduced by additional writes to the new table would be on the lambda, not the service.

Upvotes: 2

Alexander Patrikalakis
Alexander Patrikalakis

Reputation: 5195

There are a couple of approaches, but first you must understand that you cannot change the schema of an existing table. To get a different schema, you have to create a new table. You may be able to reuse your existing table, but the result would be the same as if you created a different table.

  1. Lazy migration to the same table, without Streams. Every time you modify an entry in the Person table, create a new item in the Person table using UPI and not SSN as the value for the hash key, and delete the old item keyed at SSN. This assumes that UPI draws from a different range of values than SSN. If SSN looks like XXX-XX-XXXX, then as long as UPI has a different number of digits than SSN, then you will never have an overlap.
  2. Lazy migration to the same table, using Streams. When streams becomes generally available, you will be able to turn on a Stream for your Person table. Create a stream with the NEW_AND_OLD_IMAGES stream view type, and whenever you detect a change to an item that adds a UPI to an existing person in the Person table, create a Lambda function that removes the person keyed at SSN and add a person with the same attributes keyed at UPI. This approach has race conditions that can be mitigated by adding an atomic counter-version attribute to the item and conditioning the DeleteItem call on the version attribute.
  3. Preemptive (scripted) migration to a different table, using Streams. Run a script that scans your table and adds a unique UPI to each Person-item in the Person table. Create a stream on Person table with the NEW_AND_OLD_IMAGES stream view type and subscribe a lambda function to that stream that writes all the new Persons in a new Person_UPI table when the lambda function detects that a Person with a UPI was changed or when a Person had a UPI added. Mutations on the base table usually take hundreds of milliseconds to appear in a stream as stream records, so you can do a hot failover to the new Person_UPI table in your application. Reject requests for a few seconds, point your application to the Person_UPI table during that time, and re-enable requests.

Upvotes: 42

Related Questions