error with dynamo occurred (ValidationException) when calling the Query operation: Invalid KeyConditionExpression:

Question

I am a bit new to dynamodb

See error I get when trying to get the max id of my dynamodb table in python lambda function using instructions in below StackOverflow post in below link Dynamodb max value

 An error occurred (ValidationException) when calling the Query operation: Invalid KeyConditionExpression: The expression can not be empty;\"}"

see my lambda function code below

import json
import boto3

TABLE_NAME = 'user-profiles'

dynamo_DB = boto3.resource('dynamodb')    

def lambda_handler(event, context):
    user_id = event['user_id']
    email = event['email']
    bvn = event['bvn']
    password = event['password']
    phone = event['phone']
    gender = event['gender']

    output = ''


    if len(user_id) > 1 and len(password) > 5:
        try:


            table = dynamo_DB.Table(TABLE_NAME)

            values = list(table.query(
                KeyConditionExpression='',
                ScanIndexForward=False,
                Limit=1
                )
            )
            max_id = values[0]['id']
            new_id = max_id + 1

            Item = {
                'id': str(new_id),
                'profile-id': str(new_id),
                'user_id': user_id,
                'email': email,
                'bvn': bvn,
                'password': password,
                'phone': phone,
                'gender': gender
            }
            table.put_item(Item=Item)

            output += 'Data Inserted To Dynamodb Successfully'
        except Exception as e:
            output += 'error with dynamo registration ' + str(e)
            # print(output)

    else:
        output += 'invalid user or password entered, this is ' \
                  'what i received:\nusername: ' \
                  + str(user_id) + '\npassword: ' + str(password)

    return {
        "statusCode": 200,
        "body": json.dumps({
            "message": output,
        }),
    }
    # print(output)

Answer 1

You cannot query with empty KeyConditionExpression, if you need to read all records from the table you need to use scan. But you cannot use ScanIndexForward there to order records forward.

Seems like you're trying to implement primary key incrementation. I want to warn you, your solution is not really awesome, because you easily can hit a race condition.

What I would suggest: I guess you are using id as a primary key (aka partition key). it's okay. what I would do is upsert an extra record in the table, with say increment value:

increment = table.update_item(
    Key={'id': 'increment'},
    UpdateExpression='ADD #increment :increment',
    ExpressionAttributeNames={'#increment': 'increment'},
    ExpressionAttributeValues={':increment': 1},
    ReturnValues='UPDATED_NEW',
)

new_id = increment['Attributes']['increment']

This query will update the existing record with id: 'increment' and store a new incremented number in the record, if it is the very first query the record will be created with increment: 1 and subsequent calls will increment it. ReturnValues means the query will return the result after the update and you will get a new id.

put the code in place instead of where you query the last record

so your code would look like:

import json
import boto3

TABLE_NAME = 'user-profiles'

dynamo_DB = boto3.resource('dynamodb')    

def lambda_handler(event, context):
    user_id = event['user_id']
    email = event['email']
    bvn = event['bvn']
    password = event['password']
    phone = event['phone']
    gender = event['gender']

    output = ''


    if len(user_id) > 1 and len(password) > 5:
        try:


            table = dynamo_DB.Table(TABLE_NAME)

            increment = table.update_item(
                Key={'id': 'increment'},
                UpdateExpression='ADD #increment :increment',
                ExpressionAttributeNames={'#increment': 'increment'},
                ExpressionAttributeValues={':increment': 1},
                ReturnValues='UPDATED_NEW',
            )

            new_id = increment['Attributes']['increment']

            Item = {
                'id': str(new_id),
                'profile-id': str(new_id),
                'user_id': user_id,
                'email': email,
                'bvn': bvn,
                'password': password,
                'phone': phone,
                'gender': gender
            }
            table.put_item(Item=Item)

            output += 'Data Inserted To Dynamodb Successfully'
        except Exception as e:
            output += 'error with dynamo registration ' + str(e)
            # print(output)

    else:
        output += 'invalid user or password entered, this is ' \
                  'what i received:\nusername: ' \
                  + str(user_id) + '\npassword: ' + str(password)

    return {
        "statusCode": 200,
        "body": json.dumps({
            "message": output,
        }),
    }
    # print(output)

and you're good.

Extra thoughts:

And to be 100% sure that there is no race condition on incrementation, you can implement a locking mechanism this way: Before incrementing, put an extra record with id value lock and lock attribute with any value, and use ConditionExpression='attribute_not_exists(lock)'. Then make an increment and then release the lock by removing the record lock. So while the record is there the second attempt to 'make a lock' would break by the condition that attribute lock exists and throw error ConditionalCheckFailedException (you can catch the error and show to a user that the record is locked or whatever.)

Here is an example in JavaScript sorry:

module.exports.DynamoDbClient = class DynamoDbClient {
  constructor(tableName) {
    this.dynamoDb = new DynamoDB.DocumentClient();
    this.tableName = tableName;
  }

  async increment() {
    await this.lock();

    const {Attributes: {increment}} = await this.dynamoDb.update({
      TableName: this.tableName,
      Key: {id: 'increment'},
      UpdateExpression: 'ADD #increment :increment',
      ExpressionAttributeNames: {'#increment': 'increment'},
      ExpressionAttributeValues: {':increment': 1},
      ReturnValues: 'UPDATED_NEW',
    }).promise();

    await this.unlock();

    return increment;
  }

  async lock(key) {
    try {
      await this.dynamoDb.put({
        TableName: this.tableName,
        Item: {id: 'lock', _lock: true},
        ConditionExpression: 'attribute_not_exists(#lock)',
        ExpressionAttributeNames: {'#lock': '_lock'},
      }).promise();
    } catch (error) {
      if (error.code === 'ConditionalCheckFailedException') {
        throw new LockError(`Key is locked.`);
      }

      throw error;
    }
  }

  unlock() {
    return this.delete({id: 'lock'});
  }

  async delete(key) {
    await this.dynamoDb.delete({
      TableName: this.tableName,
      Key: key,
    }).promise();
  }
}

// usage

const client = new DynamoDbClient('table');
const newId = await client.increment();

...