CODEWITHSUNDEEP
pythonamazon-web-servicesflaskaws-lambdaaws-api-gateway
andrewshih
andrewshih

Reputation: 527

How to avoid timeout from AWS API Gateway and Lambda?

I have an API endpoint on AWS API Gateway with AWS Lambda (Python & Flask) to store some data from a JSON file. e.g) curl -X POST http://www.xxx.yyy/store -d @zzz.json

However, when I tried executing the API with a bigger JSON file, I encountered a timeout error. Through my investigation, the maximum timeout setting for Lambda is 300 seconds, and API Gateway is 29 seconds. The maximum timeout for Lambda 300 sec sounds fine, but 29 seconds sounds too short. What kind of things could be a solution? The JSON data can be split by id, but it needs to be sent as one file.

EDIT: Sure I can't change the number. Any suggestion to solve this problem using another technology/system design pattern? I can't change the input, though.

EDIT 2: Currently, the Lambda function has validation based on JSON scheme, parse into models, and save into database. Any suggestions?

Update - June 4, 2024

AWS has removed the 29 second API Gateway limit:
Amazon API Gateway integration timeout limit increase beyond 29 seconds

Upvotes: 4

Views: 11216

Answers (3)

tsamaya
tsamaya

Reputation: 396

Uploading files with lambdas can be tricky and a direct upload is not recommended unless the file size is under the limits.

Warning currently:

  • API Gateway has a payload limit of 10 MB
  • API Gateway has Maximum timeout of 30 s
  • Lambda has an invocation payload (request and response) limit of 6 MB

The best approach is a basically a two step process:

  1. The client app makes an HTTP request to the lambda to get an upload URL. The lambda returns a pre-signed POST URL to S3
  2. The client post the file using the pre-signed URL

API Gateway limits : https://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html Lambda limits: https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html

Upvotes: 2

Chris Williams
Chris Williams

Reputation: 35176

Is there anyway you can update your Lambda function to hand off to another process?

By decoupling you could for example do the following:

API Gateway -> Lambda (Perform any mandatory action, then store in S3 as a blob) -> S3 -> Another Lambda to process.

Upvotes: 2

ruohola
ruohola

Reputation: 24038

The timeout value cannot be increased:

Resource or operation: Integration timeout

Default quota: 50 milliseconds - 29 seconds for all integration types, including Lambda, Lambda proxy, HTTP, HTTP proxy, and AWS integrations.

Can be increased: Not for the lower or upper bounds.

Source: https://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html

Upvotes: 1

Related Questions