How to specify attributes to return from DynamoDB through AppSync

Question

I have an AppSync pipeline resolver. The first function queries an ElasticSearch database for the DynamoDB keys. The second function queries DynamoDB using the provided keys. This was all working well until I ran into the 1 MB limit of AppSync. Since most of the data is in a few attributes/columns I don't need, I want to limit the results to just the attributes I need.

I tried adding AttributesToGet and ProjectionExpression (from here) but both gave errors like:

{
  "data": {
    "getItems": null
  },
  "errors": [
    {
      "path": [
        "getItems"
      ],
      "data": null,
      "errorType": "MappingTemplate",
      "errorInfo": null,
      "locations": [
        {
          "line": 2,
          "column": 3,
          "sourceName": null
        }
      ],
      "message": "Unsupported element '$[tables][dev-table-name][projectionExpression]'."
    }
  ]
}

My DynamoDB function request mapping template looks like (returns results as long as data is less than 1 MB):

#set($ids = [])
#foreach($pResult in ${ctx.prev.result})
    #set($map = {})
    $util.qr($map.put("id", $util.dynamodb.toString($pResult.id)))
    $util.qr($map.put("ouId", $util.dynamodb.toString($pResult.ouId)))
    $util.qr($ids.add($map))
#end
{
    "version" : "2018-05-29",
    "operation" : "BatchGetItem",
    "tables" : {
        "dev-table-name": {
            "keys": $util.toJson($ids),
            "consistentRead": false
        }
    }
}

Answer 1

2023 Update:

As Harish Vemula's indicated, projections have now been implemented.

The solution to my problem is:

#set($ids = [])
#foreach($pResult in ${ctx.prev.result})
    #set($map = {})
    $util.qr($map.put("id", $util.dynamodb.toString($pResult.id)))
    $util.qr($map.put("ouId", $util.dynamodb.toString($pResult.ouId)))
    $util.qr($ids.add($map))
#end
{
    "version" : "2018-05-29",
    "operation" : "BatchGetItem",
    "tables" : {
        "dev-table-name": {
            "keys": $util.toJson($ids),
            "consistentRead": false,
            "projection": {
                "expression" : "#id, #ouId, #otherField1, #otherField2",
                "expressionNames": {
                    "#id": "id",
                    "#ouId": "ouId",
                    "#otherField1": "otherField1",
                    "#otherField2": "otherField2"
            }
        }
    }
}

I thought I would have to escape the maps inside an array, but specifying the array's name in the expression and expressionNames was all that was required to return the whole array.

Helpful links:

• Resolver mapping template reference for DynamoDB - BatchGetItem
• Resolver mapping template reference for DynamoDB - Projections
• Resolver mapping template context reference - Info
  • The selectionSetList can be used to determine which fields have been requested
• Tutorial: DynamoDB Batch Resolvers
• Expression attribute names in DynamoDB - Nested attributes
  • Note: this is for the AWS CLI, but the escaping rules appear to be the same.

---

Original Answer:

I contacted the AWS people who confirmed that ProjectionExpression is not supported currently and that it will be a while before they will get to it.

Instead, I created a lambda to pull the data from DynamoDB.

To limit the results form DynamoDB I used $ctx.info.selectionSetList in AppSync to get the list of requested columns, then used the list to specify the data to pull from DynamoDB. I needed to get multiple results, maintaining order, so I used BatchGetItem, then merged the results with the original list of IDs using LINQ (which put the DynamoDB results back in the correct order since BatchGetItem in C# does not preserve sort order like the AppSync version does).

Because I was using C# with a number of libraries, the cold start time was a little long, so I used Lambda Layers pre-JITed to Linux which allowed us to get the cold start time down from ~1.8 seconds to ~1 second (when using 1024 GB of RAM for the Lambda).

Answer 2

Updated Answer - 2023

Thanks to @HarishVemula for pointing out this AWS update - Projections can now be defined in request template such as:

{
    "version" : "2017-02-28",
    "operation" : "GetItem",
    "key" : {
        "id" : $util.dynamodb.toDynamoDBJson($ctx.args.id)
    },
    "projection" : {
     "expression" : "#author, id, title",
     "expressionNames" : { "#author" : "author"}
    }
}

Old Answer

AppSync doesn't support projection but you can explicitly define what fields to return in the response template instead of returning the entire result set.

{
  "id": "$ctx.result.get('id')",
  "name": "$ctx.result.get('name')",
   ...
}

Answer 3

AppSync now supports DynamoDB projection expressions, using which now you can specify the attributes needed in your resolver.

In your use-case, please follow below BatchGetItem documentation to pass the arguments using projection expression.

https://docs.aws.amazon.com/appsync/latest/devguide/resolver-mapping-template-reference-dynamodb.html#aws-appsync-resolver-mapping-template-reference-dynamodb-batch-get-item

Your code then would look like below, here we are specifying only author and id fields to retrieve from batch get operation

#set($ids = [])
#foreach($pResult in ${ctx.prev.result})
    #set($map = {})
    $util.qr($map.put("id", $util.dynamodb.toString($pResult.id)))
    $util.qr($map.put("ouId", $util.dynamodb.toString($pResult.ouId)))
    $util.qr($ids.add($map))
#end
{
    "version" : "2018-05-29",
    "operation" : "BatchGetItem",
    "tables" : {
        "dev-table-name": {
            "keys": $util.toJson($ids),
            "consistentRead": false,
            "projection" : {
             "expression" : "#author, id",
             "expressionNames" : {
               "#author" : "author"
              }
          }
        }
    }
}