Reputation: 1345
I am trying to retrieve all items in a dynamodb table using a query. Below is my code:
import boto.dynamodb2
from boto.dynamodb2.table import Table
from time import sleep
c = boto.dynamodb2.connect_to_region(aws_access_key_id="XXX",aws_secret_access_key="XXX",region_name="us-west-2")
tab = Table("rip.irc",connection=c)
x = tab.query()
for i in x:
print i
sleep(1)
However, I recieve the following error:
ValidationException: ValidationException: 400 Bad Request
{'message': 'Conditions can be of length 1 or 2 only', '__type': 'com.amazon.coral.validate#ValidationException'}
The code I have is pretty straightforward and out of the boto dynamodb2 docs, so I am not sure why I am getting the above error. Any insights would be appreciated (new to this and a bit lost). Thanks
EDIT: I have both an hash key and a range key. I am able to query by specific hash keys. For example,
x = tab.query(hash__eq="2014-01-20 05:06:29")
How can I retrieve all items though?
Upvotes: 41
Views: 94582
Reputation: 1367
Inspired by the other answers and comments. If all records are needed use scan
instead of query
(it requires key filters).
ref: scan, paginator('scan')
import boto3
from boto3.dynamodb.types import TypeDeserializer
session = boto3.Session()
client = session.client("dynamodb")
paginator = client.get_paginator('scan')
response_iterator = paginator.paginate(
TableName=table_name
)
data = []
for page in response_iterator:
for item in page['Items']:
# convert to python types
deserializer = TypeDeserializer()
python_data = {k: deserializer.deserialize(v) for k, v in item.items()}
# append to list
data.append(python_data['data'])
Upvotes: 3
Reputation: 1362
.scan() does not automatically return all elements of a table due to pagination of the table. There is a 1Mb max response limit Dynamodb Max response limit
Here is a recursive implementation of the boto3 scan:
import boto3
dynamo = boto3.resource('dynamodb')
def scanRecursive(tableName, **kwargs):
"""
NOTE: Anytime you are filtering by a specific equivalency attribute such as id, name
or date equal to ... etc., you should consider using a query not scan
kwargs are any parameters you want to pass to the scan operation
"""
dbTable = dynamo.Table(tableName)
response = dbTable.scan(**kwargs)
if kwargs.get('Select')=="COUNT":
return response.get('Count')
data = response.get('Items')
while 'LastEvaluatedKey' in response:
response = kwargs.get('table').scan(ExclusiveStartKey=response['LastEvaluatedKey'], **kwargs)
data.extend(response['Items'])
return data
Upvotes: 14
Reputation: 2906
I ran into this error when I was misusing KeyConditionExpression
instead of FilterExpression
when querying a dynamodb table.
KeyConditionExpression
should only be used with partition key or sort key values.
FilterExpression
should be used when you want filter your results even more.
However do note, using FilterExpression
uses the same reads as it would without, because it performs the query based on the keyConditionExpression
. It then removes items from the results based on your FilterExpression
.
Source Working with Queries
Upvotes: 3
Reputation: 1
This is how I do a query if someone still needs a solution:
def method_name(a, b)
results = self.query(
key_condition_expression: '#T = :t',
filter_expression: 'contains(#S, :s)',
expression_attribute_names: {
'#T' => 'your_table_field_name',
'#S' => 'your_table_field_name'
},
expression_attribute_values: {
':t' => a,
':s' => b
}
)
results
end
Upvotes: -5
Reputation: 1131
I'm on groovy but it's gonna drop you a hint. Error :
{'message': 'Conditions can be of length 1 or 2 only'}
is telling you that your key condition can be length 1 -> hashKey only, or length 2 -> hashKey + rangeKey. All what's in a query on a top of keys will provoke this error. The reason of this error is: you are trying to run search query but using key condition query. You have to add separate filterCondition to perform your query. My code
String keyQuery = " hashKey = :hashKey and rangeKey between :start and :end "
queryRequest.setKeyConditionExpression(keyQuery)// define key query
String filterExpression = " yourParam = :yourParam "
queryRequest.setFilterExpression(filterExpression)// define filter expression
queryRequest.setExpressionAttributeValues(expressionAttributeValues)
queryRequest.setSelect('ALL_ATTRIBUTES')
QueryResult queryResult = client.query(queryRequest)
Upvotes: 13
Reputation: 1345
Ahh ok, figured it out. If anyone needs:
You can't use the query method on a table without specifying a specific hash key. The method to use instead is scan. So if I replace:
x = tab.query()
with
x = tab.scan()
I get all the items in my table.
Upvotes: 65