CODEWITHSUNDEEP
amazon-web-servicesaws-lambdaamazon-dynamodb
misantronic
misantronic

Reputation: 1211

AWS DynamoDB and Lambda: Scan optimizations / performance

To store api-gateway websocket-connections, I use a dynamoDB-table. When posting to stored connections, I retrieve the connection in a lambda-function via:

const dynamodb = new DynamoDB.DocumentClient();
const { Items, Count } = await dynamodb.scan({ TableName: 'Websocket' }).promise();

// post to connections

This is not really fast; the query takes around 400 - 800ms which could be better in my opinion. Can I change something on my implementation or is there maybe another aws-service which is better for storing these tiny infos about the websocket-connection (its really just a small connection-id and a user-id)?

Upvotes: 0

Views: 274

Answers (1)

best wishes
best wishes

Reputation: 6634

It has nothing to do with dynamodb, if you do a scan on any database which reads from disk, it will take time and money from your pocket.

You can use any of below solution to achieve what you are doing.

  1. Instead of storing all the websocket ids as separate row, consider having single record in which ids are stored, so that you can do a single query (not scan) and proceed.

    Cons:

    a. multiple writes to same row will result in race condition. and few reads might get lost, you can use conditional write to update record to solve this problem (have a always increasing version, and update the record only if version in db = version you read from db)

    b. There is a limit on size of single document in dynamodb. As of now it is 400kb.

  2. Store websocket id as separate row but group them by different keys, and create secondary index on these keys. Store the keys in a single row. While doing a fetch first get all relevant groups, and then query (not scan) all the items of that group. It will not exactly solve your problem but you can do interesting things like, let's say there are 10 groups, every second, messages for 1 groups are sent. this will make sure that load on your message sending infrastructure is also balanced. And you can keep incrementing number of groups as user increases.

  3. Keep the ids in a cache like aws elastic cache and add/remove ids as new entries are made in dynamodb by using aws lambda and dyanmodb streams. It will make sure that you reads are fast. At the same time if cache goes down you can use dynamodb to populate it again by doing scan on dynamodb.

    Cons:

    a. Extra component to maintain.

Upvotes: 1

Related Questions