CODEWITHSUNDEEP
pythonbotoamazon-dynamodb
dlaser
dlaser

Reputation: 1345

Retrieve all items from DynamoDB using query?

I am trying to retrieve all items in a dynamodb table using a query. Below is my code:

import boto.dynamodb2
from boto.dynamodb2.table import Table
from time import sleep

c    = boto.dynamodb2.connect_to_region(aws_access_key_id="XXX",aws_secret_access_key="XXX",region_name="us-west-2")

tab  = Table("rip.irc",connection=c)

x    = tab.query()

for i in x:
    print i
    sleep(1)

However, I recieve the following error:

ValidationException: ValidationException: 400 Bad Request
{'message': 'Conditions can be of length 1 or 2 only', '__type': 'com.amazon.coral.validate#ValidationException'}

The code I have is pretty straightforward and out of the boto dynamodb2 docs, so I am not sure why I am getting the above error. Any insights would be appreciated (new to this and a bit lost). Thanks

EDIT: I have both an hash key and a range key. I am able to query by specific hash keys. For example,

x = tab.query(hash__eq="2014-01-20 05:06:29")

How can I retrieve all items though?

Upvotes: 41

Views: 94570

Answers (6)

Dimo Boyadzhiev
Dimo Boyadzhiev

Reputation: 1367

Inspired by the other answers and comments. If all records are needed use scan instead of query (it requires key filters).

ref: scan, paginator('scan')

import boto3
from boto3.dynamodb.types import TypeDeserializer

session = boto3.Session()
client = session.client("dynamodb")

paginator = client.get_paginator('scan')

response_iterator = paginator.paginate(
    TableName=table_name
)

data = []
for page in response_iterator:
    for item in page['Items']:
        # convert to python types
        deserializer = TypeDeserializer()
        python_data = {k: deserializer.deserialize(v) for k, v in item.items()}

        # append to list
        data.append(python_data['data'])

Upvotes: 3

Ethan Harris
Ethan Harris

Reputation: 1362

.scan() does not automatically return all elements of a table due to pagination of the table. There is a 1Mb max response limit Dynamodb Max response limit

Here is a recursive implementation of the boto3 scan:

import boto3
dynamo = boto3.resource('dynamodb')


def scanRecursive(tableName, **kwargs):
        """
        NOTE: Anytime you are filtering by a specific equivalency attribute such as id, name 
        or date equal to ... etc., you should consider using a query not scan

        kwargs are any parameters you want to pass to the scan operation
        """
        dbTable = dynamo.Table(tableName)
        response = dbTable.scan(**kwargs)
        if kwargs.get('Select')=="COUNT":
            return response.get('Count')
        data = response.get('Items')
        while 'LastEvaluatedKey' in response:
            response = kwargs.get('table').scan(ExclusiveStartKey=response['LastEvaluatedKey'], **kwargs)
            data.extend(response['Items'])
        return data

Upvotes: 14

Dylan w
Dylan w

Reputation: 2906

I ran into this error when I was misusing KeyConditionExpression instead of FilterExpression when querying a dynamodb table.

KeyConditionExpression should only be used with partition key or sort key values. FilterExpression should be used when you want filter your results even more.

However do note, using FilterExpression uses the same reads as it would without, because it performs the query based on the keyConditionExpression. It then removes items from the results based on your FilterExpression.

Source Working with Queries

Upvotes: 3

Adithya
Adithya

Reputation: 1

This is how I do a query if someone still needs a solution:

def method_name(a, b)
    results = self.query(
      key_condition_expression: '#T = :t',
      filter_expression: 'contains(#S, :s)',
      expression_attribute_names: {
        '#T' => 'your_table_field_name',
        '#S' => 'your_table_field_name'
      },
      expression_attribute_values: {
        ':t' => a,
        ':s' => b
      }
    )
    results
  end

Upvotes: -5

Vadim
Vadim

Reputation: 1131

I'm on groovy but it's gonna drop you a hint. Error :

{'message': 'Conditions can be of length 1 or 2 only'}

is telling you that your key condition can be length 1 -> hashKey only, or length 2 -> hashKey + rangeKey. All what's in a query on a top of keys will provoke this error. The reason of this error is: you are trying to run search query but using key condition query. You have to add separate filterCondition to perform your query. My code

    String keyQuery = " hashKey = :hashKey and rangeKey between :start and :end "
    queryRequest.setKeyConditionExpression(keyQuery)// define key query
    String filterExpression = " yourParam = :yourParam "
    queryRequest.setFilterExpression(filterExpression)// define filter expression
    queryRequest.setExpressionAttributeValues(expressionAttributeValues)
    queryRequest.setSelect('ALL_ATTRIBUTES')
    QueryResult queryResult = client.query(queryRequest)

Upvotes: 13

dlaser
dlaser

Reputation: 1345

Ahh ok, figured it out. If anyone needs:

You can't use the query method on a table without specifying a specific hash key. The method to use instead is scan. So if I replace:

x    = tab.query()

with

x    = tab.scan()

I get all the items in my table.

Upvotes: 65

Related Questions