CODEWITHSUNDEEP
node.jsamazon-web-servicesamazon-dynamodbaws-lambda
Kees van Lierop
Kees van Lierop

Reputation: 1013

DynamoDB scan leaves valid item out

In a DynamoDB table, I have an item with the following scheme:

{
    id: 427,
    type: 'page',
    ...other_data
}

When querying on primary index (id), I get the item returned as expected.

With a scan operation inside AWS DynamoDB web app to get all items with type page, 188 items including this missing item are returned. However, performing this scan operation inside Lambda with the AWS SDK, only 162 items are returned. Part of the code looks like:

const params = { 
    TableName: <my-table-name>,
    FilterExpression: '#type = :type',
    ExpressionAttributeNames: { '#type': 'type' },
    ExpressionAttributeValues: { ':type': 'page' }
};

dynamodb.scan(params, (error, result) => {
    if (error) {
      console.log('error', error);
    } else {
      console.log(result.Items); // 162 items
    }
});

What is missing here?

Upvotes: 3

Views: 1837

Answers (1)

Daniel Conde Marin
Daniel Conde Marin

Reputation: 7742

This will probably be the case of your result data set exceeding the limit 1MB:

If the total number of scanned items exceeds the maximum data set size limit of 1 MB, the scan stops and results are returned to the user as a LastEvaluatedKey value to continue the scan in a subsequent operation. The results also include the number of items exceeding the limit. A scan can result in no table data meeting the filter criteria.

Check on the result for the LastEvaluatedKey field and use it for the next scan operation passing it as ExclusiveStartKey

Upvotes: 8

Related Questions