CODEWITHSUNDEEP
amazon-web-servicesaws-lambdaamazon-redshift
user8506442
user8506442

Reputation: 11

Timeout in lambda function after 15 minutes

I have written a function which queries data and then I process that data and call two external API's. My function works fine if the number of records are 2000, but more than that causes timeout error after 900 seconds. I have allocated 4GB for this fucntion. What else can be done in this case?

Upvotes: 1

Views: 3251

Answers (4)

Shawn
Shawn

Reputation: 9402

You could use the initial lambda execution to trigger other asynchronous lambda calls. You would loop through your 2000 records and for each one trigger another lambda, passing in details about the record to be processed. Each asynchronously-triggered lambda would process just the single record it got sent. That way you essentially process records in parallel instead of in a serial fashion.

These resources explain things a bit more:

With async invocation, your initial lambda does little more than loop through records and trigger async lambda calls for each record. You will need to think about concurrency to ensure you don't get throttled by having too many lambda executing concurrently.

Upvotes: 0

rok
rok

Reputation: 2765

Lambda is great and super-easy to use, but you have a time limit of 15 that you can not increase in any way. You also have a limit of 10GB of memory (CPU is scaled accordingly), so if you are thinking of increasing performances, take this in mind. I had the same issue and I am moving to Fargate, where you can define a task which run a docker container uploaded to ECR. You have no timeout, you can have multi-CPU environments and you can invoke the task with a lambda. It's a similar approach to what @Paolo described, look here for differences between the two services.

Upvotes: 1

Paolo
Paolo

Reputation: 25989

If you have a monolithic application that you need to run serverless and requires an execution time greater than 15 minutes, you could consider using ECS instead:

  1. Create a Docker image with your function
  2. Upload the Docker image to ECR
  3. Create an ECS Task Definition to run the container image
  4. Run an ECS task

Upvotes: 2

Tiisetso Tjabane
Tiisetso Tjabane

Reputation: 2106

Looks like the maximum time limit for lambda is 15 min, from this AWS Lambda Time limit

Try to redesign your solution to be more efficient, you can make the two API calls concurrent and use batchgets or parallel scans here is a good guide Best Practices for Querying and Scanning Data

Upvotes: 0

Related Questions