CODEWITHSUNDEEP
javascriptnode.jsamazon-s3aws-lambda
Virtually Nick
Virtually Nick

Reputation: 368

NodeJS on Lambda Using S3.getObject

So, I'm banging my head against the wall trying to figure out what in the world I am doing wrong here. I'm writing a Lambda function using NodeJS 6.10. The function is supposed to retrieve several images from one of my S3 buckets and process them. I am embracing the asynchronous nature of JavaScript, or trying to, anyway, and allowing these to happen concurrently and then wait and do something after all of them have completed. I've distilled the code here down to the bare minimum I can write and still reproduce the problem I'm seeing. The problem is that none of the images are ever retrieved - in fact, there's no indication that the Lambda function ever actually tries to retrieve them, and no indication that the callbacks are ever fired.

Here's the code:

console.log('Executing testImg method.');
var aws = require('aws-sdk');
var s3 = new aws.S3();

var imgData = {};

exports.handler = function(event, context, callback) {

    var imgs = ['img_1', 'img_2', 'img_3', 'img_4', 'img_5', 'img_6', 'img_7', 'img_8'];
    var imgPromises = [];
    console.log('Grabbing images: ' + imgs);
    imgs.forEach(function(img) {
        console.log(img);
        var myPromise = s3.getObject({
            Bucket: 'photoBucket',
            Key: 'images/' + img + '.png'
        }).promise();
        imgPromises.push(myPromise);
        myPromise.then(function(result) {
            console.log('Got ' + img);
            imgData[img] = result.Body;
        });
    });
    Promise.all(imgPromises).then(function() {
        console.log(imgData);
        context.succeed();
    });

}

And here is the output from Lambda:

START RequestId: <UUID> Version: $LATEST
2017-05-21T18:53:31.187Z    <UUID>  Grabbing images: img_1,img_2,img_3,img_4,img_5,img_6,img_7,img_8
2017-05-21T18:53:31.187Z    <UUID>  img_1
2017-05-21T18:53:31.625Z    <UUID>  img_2
2017-05-21T18:53:31.648Z    <UUID>  img_3
2017-05-21T18:53:31.706Z    <UUID>  img_4
2017-05-21T18:53:31.707Z    <UUID>  img_5
2017-05-21T18:53:31.708Z    <UUID>  img_6
2017-05-21T18:53:31.766Z    <UUID>  img_7
2017-05-21T18:53:31.767Z    <UUID>  img_8
END RequestId: <UUID>
REPORT RequestId: <UUID>    Duration: 10002.24 ms   Billed Duration: 10000 ms   Memory Size: 128 MB Max Memory Used: 80 MB  
2017-05-21T18:53:41.173Z <UUID> Task timed out after 10.00 seconds

As you can see from the output, the console.log(img) line gets run where it prints out the name of the image. However, none of the code inside the individual .then() blocks gets run (e.g. console.log('Got ' + img)), and none of the code in the final Promise.all block gets run (printing out the entire array, and calling the succeed function to terminate the function.

Also, a few other things I have tried:

I could chain them all together so that they each happen synchronously, but 1) it's hard to do this in a dynamic fashion (not knowing the number of files I'm going to need to get ahead of time), and 2) this seems to go against everything that Async JS is about these days.

Any hints would be greatly appreciated.

Upvotes: 1

Views: 1683

Answers (1)

Virtually Nick
Virtually Nick

Reputation: 368

Okay, so, Mark B's comment about S3 needing to access public Internet resources was spot on, and here are the three possible solutions:

  • Move Lambda function to a public subnet that has a route to the Internet via an IGW. I didn't really want to go this route because there's no reason to make the instances public, but it is an option.
  • Configure a NAT Gateway (formerly NAT Instance) and configure the route table to allow traffic out to the Internet endpoints via that method. Unfortunately NAT gateways incur cost, and I didn't want to run up a monthly bill.
  • The one I ended up going with was to add a S3 Endpoint to my VPC inside my private subnets, then configure the route tables, security groups, and policies such that the items inside the private subnet could access that S3 endpoint. This works perfectly.

Thanks, again, Mark, for the nudge in the right direction.

Upvotes: 0

Related Questions