CODEWITHSUNDEEP
amazon-web-servicesaws-lambdaaws-api-gatewayaws-vpc
Lucas Kauer
Lucas Kauer

Reputation: 308

API Gateway + Lambda extremely slower response time

I'm creating the backend of my application using the API Gateway with Lambda functions and I'm having problems with the response time of the requests.

It's already known that the Lambda functions have the infamous "cold start" and ok, we already accept that. But the problem I'm having seems to be is a new cold start, this time by the API Gateway. And is not a few ms of standby time, it's seconds (around 12-15 seconds). Oh god, this is a big problem...

This response delay occurs for 12-15 seconds on the first request and after some inactivity (approximately 1 hour).

My question is: what could be causing this delay and how to fix it?

More information:
My lambda function is configured to run on a VPC.

CloudWatch logs from API Gateway: 

(01) Extended Request Id: XXXXX=
(02) Verifying Usage Plan for request: XXXXX. API Key: API Stage: XXXXX
(03) API Key authorized because method 'GET /XXXXX' does not require API Key. Request will not contribute to throttle or quota limits
(04) Usage Plan check succeeded for API Key and API Stage XXXXX/v1
(05) Starting execution for request:
(06) HTTP Method: GET, Resource Path:
(07) Method request path:
(08) Method request query string:
(09) Method request headers:
(10) Method request body before transformations:
(11) Endpoint request URI:
(12) Endpoint request headers:
(13) Endpoint request body after transformations:
(14) Sending request to XXXXX
(15) Received response. Integration latency: 14497 ms
(16) Endpoint response body before transformations:
(17) Endpoint response headers:
(18) Method response body after transformations:
(19) Method response headers:
(20) Successfully completed execution
(21) Method completed with status: 200
(22) AWS Integration Endpoint RequestId :
(23) X-ray Tracing ID :

Upvotes: 4

Views: 17314

Answers (2)

Mrk Fldig
Mrk Fldig

Reputation: 4496

Update 14/12/19:

AWS have introduces provisioned Lambda : https://aws.amazon.com/blogs/aws/new-provisioned-concurrency-for-lambda-functions/

So few things to bear in mind here, a cold start occurs when the container Lambda runs in is effectively "decommissioned" - what that means is AWS infrastructure has dropped it from being "ready" to "nobody's really using this, let's shelve it".

Lambda's outside a VPC can have a cold start time up to 6 seconds, INSIDE a VPC you can be looking at anywhere up to 12 seconds PER CONTAINER, so just because you have one Lambda instance warm, if two people hit that endpoint at the same time then the 2nd person is going to get a cold start.

So as Mr Dashmug right suggests having a scheduled function to warm up your lambda's is the easy way, now one thing to remember is that your function would probably warm 1 container, if you're expecting hundreds of requests a second you need to keep X number of containers warm.

For an example of how to make this easy you can look at this - it's a plugin for the serverless framework that does exactly what you're looking for.

Essentially you need a function that's going to make X number of concurrent requests per endpoint - be aware this has a cost although you can keep a pretty decent microservice warmped up like this for less than $30 a month.

Personally, I think cold starts are overegged - sure customers can occasionally suffer a slow response, but if your API has relatively stable traffic then I really wouldn't worry your customers will keep the right number of Lambda's warm, if it's prone to spikes then it's worth warming them up.

Think about this, my average request time for API's I work on is < 400ms - so I'd need 2 requests a second, 120 a minute, 7200 an hour to even start needing two containers all the time - if you have something like app where people login, then call an api endpoint for homescreen you could do something as simple as Login->SNS fire a warmup event to the next endpoint.

Basically if you know the flow that your consumer is going to call the api, you can proactively warmup endpoints depending on the previous one called.

Upvotes: 5

Noel Llevares
Noel Llevares

Reputation: 16087

API Gateway has no cold starts, AFAIK.

That delay after one hour of inactivity is still Lambda's cold start.

To prevent that, you can create a CloudWatch Scheduled Event to keep calling your Lambda (e.g. every 5 minutes) to avoid inactivity and to reduce cold starts.

This is less of a problem once you are in production and your traffic is already high so there is less inactivity.

Upvotes: 1

Related Questions