Generate synthetic partition keys for a DynamoDB table

Question

I'm sure this is a trivial and common question with regards to DynamoDB but Google or AWS docs didn't help much.

My question is simply:

What is the pattern to generate partition keys?

In an RDBMS, this is pretty straightforward e.g. with Identity columns (MSSQL) or sequences (Postgres). However, when it comes to DynamoDB (I guess this is applicable to any NoSQL database) it's not clear how to generate the partition keys - should it be a GUID or a sequence maintained in the table itself?

Here is an example schema from AWS docs - it uses incrementing numbers as the partition key. But I can't think of a way to generate non-overalapping sequential numbers at application level (as opposed to at a database level that provides atomic transactions which DynamoDB doesn't)

Answer 1

Overall, your question is how to generate sequence numbers (like Oracle Sequence) in space like DynamoDB. I had similar challenges and below is so far better approach I found utlizing AWS serverless approach...

1. Setup a sequence number table in DynamoDB

Create a simple table e.g. order-number-sequence that has a partition key (id) and a sequence number attribute (sequence_number).

{“id”: “100”,“sequence_number”: 250000}

I set id as type string with a value “100”. This could be set as a number or any allowed type for the partition key field. The sequence_number is a numeric attribute that will act as the atomic number and in this example, I set 250000 as its initial value.

2. Create an API

Create a very minimal API with a POST method that when invoked will return the next sequence number from the above table. Use the API Gateway’s built-in integration for DynamoDB. Settings are needed for configuring the DynamoDB integration with API Gateway and to perform the UpdateItem action on DynamoDB

3. Configure request mapping template

Create a new mapping template for content type application/json. This is the most important part of this setup. We don’t need any complex scripting for this, so hopefully it should be simple enough to understand. This template contains the parameters and the payload for the UpdateItem action (required while setup API for DynamoDB integration)

 {  “TableName”: “order-sequence-number”,  
“Key”: {    “id”: {      “S”: “100”    }  },  
“ExpressionAttributeValues”: {    “:val”: {      “N”: “1”    }  },  “UpdateExpression”: “SET sequence_number = sequence_number + :val”,  “ReturnValues”: “UPDATED_NEW”}

4. Test API

We need to take just the sequence_number and its value from the response which is returned from DynamoDB’s UpdateItem.

{“sequence_number”: “$input.path(‘$.Attributes.sequence_number.N’)”}

With the above settings, if we now test the API, we should see the response as below:

{“sequence_number”: “250006”}

I hope above approach help...

Answer 2

Almost certainly, you want the partition key to be something which you know to be unique without reference to the existing data, either because its uniqueness is assumed in your business logic (like the serial number of a product or the email address of a user) or because it's universally unique (most simply by being a GUID, a timestamp, or some other one-time-only property).

Generating sequential unique identifiers is actually pretty hard to do at scale, and most RDBMS implementations are actually not as rigid as you might expect at scale (they generally make no guarantees that numbers won't be skipped, for instance, only that they'll be unique and monotonically increasing). If you think you need to use an incrementing id in your table, think very carefully about why and whether you could avoid doing so.

If you actually do need an incrementing numeric partition key, you'll need to have a service in your application which generates it for you. You could implement this as a separate dynamodb table containing only one row, with one "counter value" field which you increment using an ADD update expression. Scaling this implementation would be problematic.