Numpy in AWS lambda function

Question

I have installed pandas, numpy, s3fs and created zip file to add as layer. added that zip file s3 bucket. But unable to find what is actual version it needs. Using Python 3.7.9. Tried with manylinux wheel file but getting error not supporting to system. I want to work with pandas for reading of multiple csv files from s3 bucket which is of large size (>2GB). Please let me know which version I need to use. These concept and code working with local.

I am working on windows. Not using dockers. Simply installing packages, creating zip file, uploaded to S3 bucket. Adding that link to lambda layer. Then testing code but facing version issue. Let me know if these flow is wrong.

Answer 1

The best way to create lambda layers is using docker as described in AWS docs. Thus in your case the you would do (commands for linux):

1. Create empty folder, e.g. mylayer.

2. Go to the folder and create requirements.txt file with the content of

numpy
pandas
s3fs

3. Run the following docker command for python3.8:

docker run -v "$PWD":/var/task "lambci/lambda:build-python3.8" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.8/site-packages/; exit"

4. Create the zip layer:

zip -9 -r mylayer.zip python 

5. Create lambda layer based on mylayer.zip in the AWS Console. Don't forget to specify Compatible runtime to python3.8.

6. Add the the layer created in step 5 to your function.

Answer 2

You can have a look at this git repo which not only provides a layer that includes numpy but also has instructions on how to create such a layer. You can probably find whatever you need in there.