CODEWITHSUNDEEP
amazon-web-servicesamazon-dynamodb
Jonathan Smith
Jonathan Smith

Reputation: 2599

AWS DynamoDB Max Item size

I am trying to create a simple logging system using DynamoDB. Using this simple table structure:

{
 "userName": "Billy Wotsit",
 "messageLog": [
   {
    "date": "2022-06-08 13:17:03",
    "messageId": "j2659afl32btc0feqtbqrf802th296srbka8tto0",
    "status": 200
   },
   {
    "date": "2022-06-08 16:28:37.464",
    "id": "eb4oqktac8i19got1t70eec4i8rdcman6tve81o0",
    "status": 200
   },
   {
    "date": "2022-06-09 11:54:37.457",
    "id": "m5is9ah4th4kl13d1aetjhjre7go0nun2lecdsg0",
    "status": 200
   }
  ]
}

It is easily possible that the number of items in the message log will run into the thousands.

According to the documentation an "item" can have a maximum size of 400kB which severly limits the maximum number of log elements that can be stored.

What would be the correct way to store this amount of data without resorting to a more traditional SQL-approach (which is not really needed)

Upvotes: 0

Views: 1100

Answers (1)

Hyo Byun
Hyo Byun

Reputation: 1276

Some information on use cases for your data would help. My primary questions would be

  1. How often do you need to update/ append logs to an entry?
  2. How do you need to retrieve the logs? Do you need all of them? Do you need them per user? Will you filter by time?

Without knowing any more info on your data read/write patterns, a better layout would be to have each log entry be an item in the DB:

  1. The username can be the partition key
  2. The log date can be the sort key.
  3. (optional) If you need to ever retrieve by message id, you can create a secondary index. (Or vice versa with the log date)

With the above structure, you can trivially append log entries & retrieve them efficiently assuming you've set your sort key / indexes appropriately to your use case. Focus on understanding how the sort key and secondary indexes optimize finding your entries in the DB. You want to avoid scanning through the entire database looking for your items. See Core Components of Amazon DynamoDB for more info.

Upvotes: 2

Related Questions