CODEWITHSUNDEEP
amazon-web-servicesamazon-s3aws-lambda
Brooks
Brooks

Reputation: 7380

OutOfMemoryError when creating AmazonS3Client in Lambda

I have an AWS Lambda function, configured with only 128MB of memory, is triggered by SNS (which is itself triggered by S3) and will download the file from S3.

In my function, I have the following:

public class LambdaHandler {

    private final AmazonS3Client s3Client = new AmazonS3Client();

    public void gdeltHandler(SNSEvent event, Context context) {
        System.out.println("Starting");
        System.out.println("Found " + eventFiles.size() + " event files");
    }

I've commented out and excluded from this post all of the logic because I am getting an OutOfMemoryError which I have isolated to the creation of the AmazonS3Client object. When I take that object out, I don't get the error. The exact above code results in the OutOfMemoryError.

I assigned 128MB of memory to the function, is that really not enough to simply grab the credentials and instantiate the AmazonS3Client object?

I've tried giving the AmazonS3Client constructor

new EnvironmentVariableCredentialsProvider()

as well as

new InstanceProfileCredentialsProvider()

with similar results.

Does the creation of the AmazonS3Client object simply require more memory?

Below is the stack trace:

Metaspace: java.lang.OutOfMemoryError java.lang.OutOfMemoryError: Metaspace at com.fasterxml.jackson.databind.deser.BeanDeserializerBuilder.build(BeanDeserializerBuilder.java:347) at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.buildBeanDeserializer(BeanDeserializerFactory.java:242) at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:143) at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:409) at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:358) at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:265) at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:245) at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:143) at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:439) at com.fasterxml.jackson.databind.ObjectReader._prefetchRootDeserializer(ObjectReader.java:1588) at com.fasterxml.jackson.databind.ObjectReader.(ObjectReader.java:185) at com.fasterxml.jackson.databind.ObjectMapper._newReader(ObjectMapper.java:558) at com.fasterxml.jackson.databind.ObjectMapper.reader(ObjectMapper.java:3108)

When I try providing the InstanceProfileCredentialsProvider or EnvironmentVariableCredentialsProvider, I get the following stack trace:

Exception in thread "main" java.lang.Error: java.lang.OutOfMemoryError: Metaspace at lambdainternal.AWSLambda.(AWSLambda.java:62) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at lambdainternal.LambdaRTEntry.main(LambdaRTEntry.java:94) Caused by: java.lang.OutOfMemoryError: Metaspace at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at lambdainternal.EventHandlerLoader$PojoMethodRequestHandler.makeRequestHandler(EventHandlerLoader.java:421) at lambdainternal.EventHandlerLoader.getTwoLengthHandler(EventHandlerLoader.java:777) at lambdainternal.EventHandlerLoader.getHandlerFromOverload(EventHandlerLoader.java:802) at lambdainternal.EventHandlerLoader.loadEventPojoHandler(EventHandlerLoader.java:888) at lambdainternal.EventHandlerLoader.loadEventHandler(EventHandlerLoader.java:740) at lambdainternal.AWSLambda.findUserMethodsImmediate(AWSLambda.java:126) at lambdainternal.AWSLambda.findUserMethods(AWSLambda.java:71) at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:219) at lambdainternal.AWSLambda.(AWSLambda.java:60) ... 3 more START RequestId: 58837136-483e-11e6-9ed3-39246839616a Version: $LATEST END RequestId: 58837136-483e-11e6-9ed3-39246839616a REPORT RequestId: 58837136-483e-11e6-9ed3-39246839616a Duration: 15002.92 ms Billed Duration: 15000 ms Memory Size: 128 MB Max Memory Used: 50 MB
2016-07-12T14:40:28.048Z 58837136-483e-11e6-9ed3-39246839616a Task timed out after 15.00 seconds

EDIT 1 If I increase the memory allocated to the function to even 192MB, it works just fine, though strangely enough, reports only using 59MB of memory in the cloudwatch logs. Am I simply losing the rest of the memory?

Upvotes: 38

Views: 23232

Answers (4)

Jim Tough
Jim Tough

Reputation: 15230

I use a tactic that helps for Java-based lambdas. Any class resources that only need a single (reusable) instance can be declared as static class members, and initialized inside a static initializer block. When the lambda creates a new instance of the class to handle an execution, those expensive resources are already initialized. Here is a simple example:

package com.mydomain.myapp.lambda.sqs;

import com.amazonaws.services.lambda.runtime.events.SQSEvent;
import com.amazonaws.services.sns.AmazonSNS;
import com.amazonaws.services.sns.AmazonSNSClientBuilder;
import com.amazonaws.services.sqs.AmazonSQS;
import com.amazonaws.services.sqs.AmazonSQSClientBuilder;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.sql.Connection;
import java.sql.SQLException;
import java.util.Objects;

public class MyLambdaFunctionHandler {

    private static final Logger LOGGER = LoggerFactory.getLogger(MyLambdaFunctionHandler.class);

    // These values come from the 'Environment' property for the lambda, defined in template.yaml
    private static final String ENV_NAME = System.getenv("ENV_NAME");

    // Declare these as static properties so they only need to be created once,
    // rather than on each invocation of the lambda handler method that uses them
    private static final ObjectMapper OBJECT_MAPPER;
    private static final AmazonSNS SNS;
    private static final AmazonSQS SQS;

    static {
        LOGGER.info("static initializer | START");
        Objects.requireNonNull(ENV_NAME, "ENV_NAME cannot be null");
        OBJECT_MAPPER = new ObjectMapper();
        SNS = AmazonSNSClientBuilder.defaultClient();
        SQS = AmazonSQSClientBuilder.defaultClient();
        LOGGER.info("static initializer | END");
    }

    public MyLambdaFunctionHandler() {
        LOGGER.info("constructor invoked");
    }

    public void handlerMethod(SQSEvent event) {
        LOGGER.info("Received SQSEvent with {} messages", event.getRecords().size());
        event.getRecords().forEach(message -> handleOneSQSMessage(message));
    }

    private void handleOneSQSMessage(SQSEvent.SQSMessage message) {
        // your SQS message handling code here...
    }

}

The properties I declared as static will stay in memory until the lambda instance is destroyed by AWS.

This isn't how I would normally write Java code. Lambda-based code is treated differently, so I think it is OK to break some traditional patterns here.

Upvotes: 3

user3311298
user3311298

Reputation: 375

One way to reduce cold start is setting the memory to 1536 mb and the timeout to 15 min. This will give dedicated host to run only your lambda instead of running your lambda on shared host + when a new instance has to be started, it will copy the code from cache on the host rather than copying from S3.

This though will be more expensive and if you don't want to do this, continue reading below.

How can I reduce my cold start times?

  1. Follow the Lambda best practices
    https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html

  2. By choosing a larger memory setting for your function
    Think of the memory as a "power" setting because it also dictates how much CPU your function will receive.

  3. By reducing the size of your function ZIP
    This likely means reducing the number of dependencies you include in your function ZIP. Java JARs can be further reduced in size using ProGuard

  4. [Java Only] Use the bytestream interface instead of the POJO interface.
    The JSON serialization libraries that Lambda uses internally can take some time to start. It will take dev work on your end, but you may be able to improve on this by using the byte stream interface along with a lightweight JSON library. Here are some links that may help: http://docs.aws.amazon.com/lambda/latest/dg/java-handler-io-type-stream.html https://github.com/FasterXML/jackson-jr

  5. [Java Only] Don't use Java 8 feature that replaces anonymous classes (lambdas, method references, constructor references, etc.)
    We've noticed internally that Java 8 Lambda-related bytecode appears to result in sub-optimal startup performance. If your code is using any Java 8 feature that replaces anonymous classes (lambdas, method references, constructor references, etc.) you may get better startup time by moving back to anonymous classes.

  6. By using a different runtime
    Different runtimes have different cold start times, and different runtime performance. While NodeJS might be better for heavy IO work, Go might be better for code that does a lot of concurrent work. Customers have done some basic benchmarks to compare language performance on Lambda, and here is a more generic comparison of different programming languages performance. There is no one-size-fits-all answer, use what makes sense for your requirements.

basic benchmarks:https://read.acloud.guru/comparing-aws-lambda-performance-of-node-js-python-java-c-and-go-29c1163c2581

generic comparison : https://benchmarksgame-team.pages.debian.net/benchmarksgame/which-programs-are-fast.html

Upvotes: 7

Pratap Vhatkar
Pratap Vhatkar

Reputation: 701

Try to increase the memory allocated to lambda from 128 to 256 MB

Upvotes: 6

Alexej Kubarev
Alexej Kubarev

Reputation: 853

I have been observing this when using AWS Java SDK within the Lambda function. It would seem when creating any of the AWS clients (Sync or Async) you may get out of Metaspace.

I believe this is due to things that the Amazon Client is performing upon instantiation, including AmazonHttpClient creation as well as dynamic loading of request handler chains (part of AmazonEc2Client#init() private method).

It is possible that the reported memory usage is for Heap itself, but may not include Metaspace. There are a few threads on AWS Forums but no responses from AWS on the matter.

Upvotes: 27

Related Questions