CODEWITHSUNDEEP
amazon-dynamodbboto3
ZZzzZZzz
ZZzzZZzz

Reputation: 1844

Boto3 dynamodb query on a table with 10Gb size

I have been trying to fetch all the records on one of my GSI and have seen that there is a option to loop through using the LastEvaluatedKey in the response only if I do a scan. I did not find a better way to use pagination using query in boto3. Is it possible to paginate using a query.

import boto3 from boto3.dynamodb.conditions import Key, Attr

dynamodb = boto3.resource('dynamodb')
res = table.query(
    TableName='myTable',
    IndexName='my-index',
    KeyConditionExpression=Key('myVal').eq(1)
)
while 'LastEvaluatedKey' in res:

   for item in res['Items']:
       print item #returns only a subset of them

Upvotes: 1

Views: 2815

Answers (1)

mootmoot
mootmoot

Reputation: 13166

The document mentioned the limit of boto3.dynamodb.table.query() : 1MB data.

You can only use Paginator.Query return iterator(which make sense).

It seems you can replace your table.query with the Paginator.Query. Try it out.

Notes : There is a catch for boto3.resource() : not all resources services are implemented. So for the dynamodb pagination generator, this is one of those case.

import boto3 
dyno_client = boto3.client('dynamodb')
paginator = dyno_client.get_paginator('query')
response_iterator = paginator.paginate(.....)

Upvotes: 2

Related Questions