Inconsistent response time of Lambda function under a VPC

Question

Recently, we have started exploring AWS lambda technologies with huge hope that we should be able to make all of our micro services to be server-less. These services are running in EC2 instances at this moment in production. So we picked one of them and tried to make it as lambda. There I am stuck by seeing the performance. I believe I am doing something seriously wrong and need your assistance to come out of the problem. We have a java application that runs on Tomcat in a EC2 instance and it fetches data from another instance under same VPC and same region. Please note that there are couple of back and forth calls between these servers. The time taken for single request to the Tomcat service is between 4 to 5 secs and that is pretty consistent. Now when we run the same service under Lambda infrastructure, the response becomes very inconsistent (5 to 25 secs).

I did not associate our VPC with the lambda in the first phase. And I thought that could be the reason for inconsistent response time, since there are couple of http requests over internet between Lambda function and one of our EC2 instances. So I associated our VPC with the Lambda but sadly seeing same response time. Below are the vpc configuration I did. 1. We have only one VPC that contains all of our EC2 instances. I associated that with the Lambda function configuration through AWS console. 2. We have 2 subnets in the VPC. Both the subnets are given in the vpc-config. 3. I have added the security group ID that is assigned to my 2nd EC2 instances where I am making the calls.

My understanding is, if I associate a VPC with a Lambda function, the function will run on a machine that is under same VPC. So we should get GBPS speed as LAN between Lambda function and associated EC2 instances. But that is not happening perhaps. Our Tomcat service gives predictable response time since they are on same VPC LAN. Please note I am measuring my functions response time directly by invoking it from client, no API gateway in between so far.

Below are the times (in milliseconds) taken by my Lambda after it's deployed for 10 subsequent calls: 16797, 17965, 10026, 5846, 7689, 5118, 4349, 9297, 4378, 11692

Where the Tomcat service (which runs the same code inside on the same data set) takes below times for 10 subsequent calls. 3597, 3568, 4009, 3808, 3585, 3859, 3576, 3657, 3444, 3440,

Please observe that the service calls to Tomcat service does not take more than 4 seconds. Does "associating a Lambda with a VPC" mean that the Lambda program runs on the same LAN as other EC2 instances lying on same VPC? Or it's a secure connection over internet?

Edit1: Below are the notes for all the points you have raised.

1. Cold Start Time: I understood the fact. And I don't have any problem with it. If Lambda is idle for 5 minutes, the next call will take more time to Cold start. But the calls I made subsequently with a interval of 2 seconds. If I discount the first call for longer cold start time, then subsequent calls are also did not take much lesser time. In fact the 2nd call took more time than first call. If I compare server based response with lambda response, it's just one third in this case. Any explanation?

2. Specified Lambda memory: I have configured the lambda to take the max memory available (i.e 1.5 GB). So this can not cause the issue.

3. ENI: This is completely new thing to me and I do not have much knowledge on it. The EC2 instance that gets call from Lambda is c3.large. And from AWS doc I see Maximum Network Interfaces given for this kind of instance is 3. Is there any way we can tune it for the performance? If we go for bigger instance then can I expect better performance?

4. Region: Both VPC and Lambda are in same region.

5. Code Initialization: There is no data base connect from Lambda. We call http endpoint from Lambda to an EC2 box that hosting few services.

Edit2: Further observations. I have created 2 test lambdas which connect to same EC2 endpoint lying in our VPC and fetch some data.

1. Lambda1: Without VPC association. Calls to EC2 endpoint once through public IP.
2. Lambda2: With VPC enabled, has access to the security group of EC2 instance. Calls to EC2 endpoint once through private (subnet) IP. Both takes similar amount of time. Only 1st run of Lambda2 (VPC enabled) takes larger time for the Cold Start. I expected the response time will go down in Lambda2 since we are bringing the client and service under a same VPC (which should be equivalent of a LAN). But that's not happening. The same java code I ran in another EC2 instance under same vpc (as a simple java application) and that takes consistently 1/3rd time than Lambda1 or Lambda2. Is there any way I can get the same response time from a Lambda? Please help.

Edit3: Here is the code I run from Lambda or Simple Java Application.

public class Lambda implements RequestStreamHandler {
@Override
public void handleRequest(InputStream inputStream, OutputStream outputStream, Context context) throws IOException {
try {
JSONObject payload = (JSONObject) new JSONParser().parse(new InputStreamReader(inputStream, "UTF-8"));
System.out.println("payload = " + payload);
outputStream.write(Request.Get("http://" + (String)payload.get("ip") + "/api/xxx").
execute().returnContent().toString()).getBytes());
} catch (ParseException e) {
e.printStackTrace();
}
}
}

ip in the payload is sent as Public IP for Lambda1 (With out VPC) and private or local ip for Lambda2 (VPC enabled) from caller.

Answer 1

Lambda with a VPC means that the Lambda runs on the same set of subnets as other EC2 instances lying on same VPC.

There are several factors responsible for variable performance of Lambda.

• Cold start:This happens when Lambda function runs for the first time or haven't been triggered around last 5 minutes. To keep Lambda Hot, you should be able to create a scheduled Lambda to trigger it around 5 mins intervals with fast exit routine in Lambda code e.g 'ping' which will call the callback immediately.

• Specified Lambda memory: If you provision 128Mb memory for a Lambda, Amazon will provision these in low end hardware which will slowdown the endpint network performance and CPU performance.

• Elastic Network Interface (ENI): The time it takes to provision an Elastic Network Interface (ENI) can also impact the API.

• If you are acessing resources in a VPC using Lambda, having Lambda in the same region and availability zone will improve performance.

• Also keep initialization code such as database connection & etc outside handler, which will be reused for subsequent call.

Answer 2

What you're experiencing are known as cold starts. This is when the underlying AWS infrastructure must be provisioned and your function brought into memory. In a VPC the issue typically is worse due to the time it takes to provision an Elastic Network Interface (ENI). Some of your subsequent calls are faster because Lambda will keep containers available and warm for a period of time. You can find more info in the AWS docs. The below link is also relevant -

https://forums.aws.amazon.com/thread.jspa?threadID=181348

Additionally, there are several projects to keep your Lambdas warm. Some good ones are found here -

https://lambdacult.com/spark https://serverless.com/blog/keep-your-lambdas-warm/