CODEWITHSUNDEEP
amazon-web-servicesaws-lambda
BBDG
BBDG

Reputation: 365

Work around for AWS Lambda 500MB /tmp storage limit

Is it possible to add temporary storage space to AWS Lambda so I can write files larger than 500 MB before I upload those to S3? Is NFS/EFS possible?

Thank you.

Upvotes: 7

Views: 3201

Answers (2)

Praveesh
Praveesh

Reputation: 1327

AWS has recently added support for attaching ephemeral storage up to 10GB to lambdas. The ephemeral storage is not shared between the invocations and is chargeable only if the size is more than 512 MB.

Ephemeral storage can be enabled through aws-cli with the following command

$ aws lambda update-function-configuration --function-name yourLambdaFunction \
              --ephemeral-storage '{"Size": 10240}'

If you are using sam-cli for managing your lambdas, the empehemeral storage can be configured in the template.yml file as given below

  yourLambdaFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: lambda_test/
      Handler: app.lambda_handler
      Runtime: python3.7
      MemorySize: 512
      EphemeralStorage:
        Size: 5120
      Timeout: 900

The size should be specified in MBs and should not exceed the maximum limit of 10GB. Also, this option is supported by sam-cli versions 1.43.0 or above. For versions below that this will through an error during the build process.

Upvotes: 3

Yeshodhan Kulkarni
Yeshodhan Kulkarni

Reputation: 2943

Nope. You shouldn't rely on it. And EFS isn't an option either.

Also, the time taken to download a file more than 500MB depends on network conditions and you won't able to predict before hand how long will it take for the function to complete.

Your lambda function must complete execution within 300 seconds.

Moreover, if multiple instances of this lambda function are invoked, you will always be constrained for storage space if all requests are going to process files larger than 500MB.

You are better off, moving this functionality outside lambda.

Upvotes: 7

Related Questions