CODEWITHSUNDEEP
node.jsamazon-web-servicesaws-lambda
Chet
Chet

Reputation: 144

Invoke a AWS Lambda function N number of times in a loop, get back responses for each lambda execution and continue rest of execution

Context: I have a lambda function that does file operations and given an input of student objects, creates pdf files for each student, zips them up and uploads them to s3.

But this array of student objects can potentially be large. 5k-10k or even more. So what i'd like to do is split the inputs by 1k students each request or so depending on the input and invoke the lambda. I've already come up with a mechanism for doing this in code. For example if input has 4.5k students, then it would split it into 5 sub arrays and I can invoke the function 5 times for each input. This will need a bit of rework on the zipping and uploading and might have to zip it all up inside s3 after all the lambda execution are completed, but that's for later.

I'd like to know how to I achieve invoking the lambda for example 5 times, wait for 5 responses I guess, and after that continue with remaining code(once its zipped and on s3 i'm sending an email with the link to download the file). Let's scale it down and fire 5 lambda calls for each element in the array.

//lambda config code

let students = [ [{"name": "Chet"}], [{"name": "Tom"}], [{"name": "Mic"}], [{"name": "Pio"}], [{"name": "Kobo"}] ]
for (stu in students) {
  const params = {
    FunctionName: "pdfZipperFunction",
    Payload: JSON.stringify(stu),
  };
  lambda.invoke(params, function(error, data) {
    if(error){
      //console.log(error)
    }
    else {
      //console.log(data.Payload.result) //success. files created and uploaded to s3
    } 
  }
}

//how do i know here that the lambda has completed executing 5 times? before i proceed further? how do I keep track?

//..continue execution rest of the code. I need to access s3, and zip all the folder that all the above 5 lambda functions uploaded files into and then send an email

Upvotes: 0

Views: 2023

Answers (1)

John Rotenstein
John Rotenstein

Reputation: 269826

Your suggested architecture is not really suitable for AWS Lambda. Having an AWS Lambda function wait for another function has no particular benefit since the service is charged per-second of execution.

A more cloud-friendly approach would be to orchestrate the work via an Amazon SQS queue:

  • Have your main process push a message into an Amazon SQS queue for each piece of work required (eg creating a PDF for one student) -- this could be done by an AWS Lambda function, or even a program that you run on an EC2 instance or perhaps your own computer
  • Configure the SQS queue to trigger an AWS Lambda function when messages are received
  • That Lambda function then looks at the message it is given (via the event parameter), performs the work and then exits

The beauty of this design is that it is highly scalable (just add more messages to the queue!) and the AWS Lambda function that processes each message can run in parallel (default limit of 1000 concurrent executions).

The Lambda function that processes messages can be configured to receive up to 10 messages per execution, or even just 1 message per execution. Either way is fine, depending on how you want your code to operate.

Upvotes: 3

Related Questions