CODEWITHSUNDEEP
amazon-web-servicesamazon-dynamodbboto3
DenCowboy
DenCowboy

Reputation: 15076

DynamoDB scan all items that meet condition

I have a DynamoDB script. I scan for item that meet a certain condition. Now it works but for tables with a lot of items it seems that it only prints some of the items that match, not all of them:

dynamodb = boto3.resource('dynamodb', region_name='eu-west-1')
table = dynamodb.Table(table)

# get items
fe = Attr('message').contains(filter)
get_response = table.scan(
    FilterExpression=fe
)

print(get_response)

But this prints not all of the items.

Upvotes: 0

Views: 1230

Answers (1)

maafk
maafk

Reputation: 6876

The scan method without pagination will (according to the docs),

A single Scan operation reads up to the maximum number of items set (if using the Limit parameter) or a maximum of 1 MB of data and then apply any filtering to the results using FilterExpression

Try using the scan paginator

dynamodb = boto3.client('dynamodb')
paginator = dynamodb.get_paginator('scan')

response_iterator = paginator.paginate(
    TableName=table,
    FilterExpression='Message = :filter',
    ExpressionAttributeValues={
        ":filter": {
            "S": "some filter"
        }
    }
)

for response in response_iterator:
    print(response)

The FilterExpression and ExpressionAttributeValues will depend on what you're looking for. check the docs for more details

Upvotes: 3

Related Questions