Lambda Calls DynamoDB DocumentClient Multiple Times through While Loop

Question

Happy Thanksgiving everyone! I am writing a lambda function which calls DynamoDB and I am running into a design/implementation issue.

I have the following code structure:

exports.handler = function(event, context, callback) {
    documentClient.batchGet(getParams, function(err, data) {
        if (err) {
            console.log(err);
        } else {
         ...
              while (1) {
                    documentClient.scan(scanParams, function(err, data)    
                    {
                        if (err) {
                            console.log(err);
                        } else {
                              ...
                              if (some condition) {
                                   break;
                              }
                         }
                 }
          }
   }

I can't call break from inside the callback of a documentClient. Also I can't pass variables from inside the documentClient's callback to outside, and then break out of the while loop. I tried creating a variable in the scope of the handler and assigning it a value inside the documentClient callback function, but once the code gets out of scope of the callback, that value is erased and the variable has its original value in the handler scope.

The reason why there is a while loop is because if the table is large scanning will take more than one trial.

Also, another issue is I want to do a callback in the same scope as the handler, not within the nested blocks like the callback function of the scan documentClient. However, I can't seem to pass any data or variable from inside the nested blocks to outside.

I searched and can't find any doc's on this problem. Thanks in advance for reading.

Answer 1

Assuming you can only use Node.js 6.x (and not allowed to use newer versions) here is the answer:

// Node.js 6.x answer:

exports.handler = function (event, context, callback) {
    documentClient.batchGet(getParams, function (err, data) {
        if (err) {
            console.log(err);
        } else {
            // ...
            runScan(scanParams, function(err, scanOutput) {
                if (err) return callback(err);
                // do something with scanOutput.
            });
        }
    });
};


function runScan(scanParams, callback) {
    documentClient.scan(scanParams, function (err, data) {
        if (err) return callback(err);
        if (some_condition) {
            var scanOutput = ...;
            return callback(null, scanOutput);
        }

        // Halting condition
        if(!data.LastEvaluatedKey) {
            return callback(new Error('End of scan'));
        }

        scanParam.ExclusiveStartKey = data.LastEvaluatedKey;
        runScan(scanParams, callback);
    });
}

Key takeaways:

• I moved the scanning functionality into its own function (runScan()).

• Instead of using a while-loop we use recursion: runScan() calls itself - but with a different ExclusiveStartKey - to scan the next batch of items.

• runScan() it an async method: it does not return value in the traditional sense. Instead it takes a callback function. If it detects an error it calls callback(err) (equivalent of throwing an exception). If it wants to "return a value" to its caller it calls callback(null, value); The caller must pass a callback function (that takes two parameters) and inspect them respectively.

• The scan ends when there is no LastEvaluatedKey. In here it is reported back to the caller by passing an Error object (an exception) as the first parameter of the callback.

• Finally, if you are allowed to use Node.js 8.x (or higher) you can use async/await which greatly simplify writing async code.