CODEWITHSUNDEEP
pythonaws-lambdaserverless-framework
Warlax56
Warlax56

Reputation: 1202

Serverless python requirements slim:true does nothing for dependency size

Brief

when using the following settings in serverless.yml

custom:
  pythonRequirements:
    dockerizePip: true
    slim: true
    zip: true

slim:true and slim:false result in the same file size. What can I do to decrease the file size of the zip file?

Full

I'm trying to get python code and it's dependencies onto an aws lambda function using the serverless framework. following the recommendations provided here to reduce the dependency size to comply with the 250MB lambda limit. in my serverless.yml file I have the following:

provider:
  name: aws
  runtime: python3.6

plugins:
  - serverless-python-requierments

custom:
  pythonRequierments:
    dockerizePip: true
    slim: true
    zip: true

functions:
...

my requierments.txt is the following:

xgboost==1.3.3
pandas==1.0.1
numpy == 1.18.5
...

I keep getting the following error:

Unzipped size must be smaller than 262144000 bytes..

which leads me to believe that my requierments.zip file is too large; the verbose output stated uploading service test.zip file to s3 271.17 MB.

I was playing around with making the file smaller, and I ran sls deploy with dockerizePip, and slim set to false for testing purpose, and I got the same result; a test.zip file of 271.17 Mb. This seemed perplexing, as I understood it slim: true would reduce the file size prior to zipping the file, but running with slim:true and slim:false resulted in the same file size. How can I make slim work?

Update1:

I tried adding a slimPattern as discussed here, within pythonRequierments

slimPatterns:
  - '**/*.dist-info/**'

but all dist info directories still exist in the zip file, and the output is the same size (271.17MB)

Update 2:

I kept everything the same from update 1, but removed the .serverless directory and requierments.zip. All the .dist-info directories remain, even though I excluded them in slimPatterns (or so I thought).

Update 3:

I tried using package exclude, doing the following:

package:
  exclude:
    -node_modules/**
    -'**/*.dist-info/**'

this did succeed in getting node_modules removed (which I should have done before), but did not succeed in changing .requierments.zip in any way. all .dist-info directories remain.

Update 4:

I followed this, and added a noDeploy field in pythonRequirements, which included setuptools and six. despite this, both ended up in my .requirements.zip, and my file size hasn't changed (269.9 Mb, the same as update 3, which was slightly smaller due to the removal of node_modules)

Upvotes: 6

Views: 3595

Answers (2)

Warlax56
Warlax56

Reputation: 1202

Really, the underlying concept behind this question is flawed. The main purpose for this question was how to squeeze large dependencies onto Lambdas size constraints. Recently, AWS released EFS support for Lambda functions, meaning you can mount volumes directly onto your lambda function. This means practically infinite storage, and no zipping requirements, meaning significantly less cold start. Also, one of these file systems can mount to multiple Lambda functions.

I won't go into insane detail, there's a lot to do, but If you want to get large dependency trees on Lambda, I recommend the following:

  • create a VPC
  • create an ec2 instance
  • create an EFS volume and mount it to your ec2 instance
  • install your dependencies to the efs instance via pip install -r requierments.txt -t /mnt/.../efs/path
  • mount that EFS instance to your lambda in the serverless.yml
  • add the modules on your EFS volume to your lambda function by appending it to path

Upvotes: 1

Sarthak Jain
Sarthak Jain

Reputation: 2096

It didn't work for me either. I ended up using Layers using the following option -

custom:
  pythonRequirements:
    slim: true
    layer: true

This reduced my deployment package size to less than 10mb. All the dependencies then go into a separate zip (~100mb) which creates a Lambda Layer. This helps reduce the Lambda cold-start problem as well.

Upvotes: 3

Related Questions