How to use AWS Lambda layer using Python?

Question

I have a simple Lambda function which is using the numpy library,
I have set up a virtual environment in my local, and my code is able to fetch and use the library locally.

I tried to use AWS Lambda's layer, and zipped the venv folder and uploaded to the layer,
Then I attached the correct layer and version to my function,
But the function is not able to fetch the library

Following is the code which works fine on local -

import numpy as np

def main(event, context):
    a = np.array([1, 2, 3])

    print("Your numpy array:")
    print(a)

Following is the venv structure which I zipped and uploaded -

I get the following error -

{
  "errorMessage": "Unable to import module 'handler': No module named 'numpy'",
  "errorType": "Runtime.ImportModuleError"
}

My Lambda deployment looks like this -

I'm trying to refer this -
https://towardsdatascience.com/introduction-to-amazon-lambda-layers-and-boto3-using-python3-39bd390add17

Answer 1

That's bit of misleading question, because you at least did not mention you use serverless. I found it going through the snapshot of you project structure you provided. That means you probably use serverless for deployment of your project within AWS provider.

Actually, there are multiple ways you can arrange lambda layer. Let's have a look at each of them.

Native AWS

Once you will navigate to Add a layer, you will find 3 options: [AWS Layers, Custom Layers, Specify an ARN;].

Specify an ARN Guys, who did all work for you: KLayers

so, you need numpy, okay. Within lambda function navigate to the layers --> create a new layer --> out of 3 options, choose Specify an ARN and as the value put: arn:aws:lambda:eu-west-1:770693421928:layer:Klayers-python38-numpy:12. It will solve your problem and you will be able to work with numpy Namespace.

Custom Layers

Choose a layer from a list of layers created by your AWS account or organization.

For custom layers the way of implementing can differ based on your requirements in terms of deployment. If are allowed to accomplish things manually, you should have a glimpse at following Medium article. I assume it will help you!

AWS Layers

As for AWS pre-build layers, all is simple.

Layers provided by AWS that are compatible with your function's runtime. Can differentiate between runtimes

For me I have list of: Perl5, SciPy, AppConfig Extension

Serverless

Within serverless things are much easier, because you can define you layers directly with lambda definition in serverless.yml file. Afterwards, HOW to define them can differ as well.

Examples can be found at: How to publish and use AWS Lambda Layers with the Serverless Framework

If you will have any questions, feel free to expand the discussion. Cheers!

Answer 2

I've seen that a few libraries like numpy and pandas don't work in Lambda when installed using pip. I have had success using the .whl package files for these libraries to create the Lambda layer. Refer to the steps below:

NOTE: These steps set up the libraries specific to the Python 3.7 runtime. If using any other version, you would need to download the .whl files corresponding to that Python version.

1. Create an EC2 instance using Amazon Linux AMI and SSH into this instance. We should create our layer in Amazon Linux AMI as the Lambda Python 3.7 runtime runs on this operating system (doc).

2. Make sure this instance has Python3 and "pip" tool installed.

3. Download the numpy .whl file for the cp37 Python version and the manylinux1_x86_64 OS by executing the below command:

$ wget https://files.pythonhosted.org/packages/d6/c6/58e517e8b1fb192725cfa23c01c2e60e4e6699314ee9684a1c5f5c9b27e1/numpy-1.18.5-cp37-cp37m-manylinux1_x86_64.whl

4. Skip to the next step if you're not using pandas. Download the pandas .whl file for the cp37 Python version and the manylinux1_x86_64 OS by executing the below command:

$ wget https://files.pythonhosted.org/packages/a4/5f/1b6e0efab4bfb738478919d40b0e3e1a06e3d9996da45eb62a77e9a090d9/pandas-1.0.4-cp37-cp37m-manylinux1_x86_64.whl

5. Next, we will create a directory named "python" and unzip these files into that directory:

        $ mkdir python
        $ unzip pandas-1.0.4-cp37-cp37m-manylinux1_x86_64.whl -d python/
        $ unzip numpy-1.18.5-cp37-cp37m-manylinux1_x86_64.whl -d python/

6. We also need to download "pytz" library to successfully import numpy and pandas libraries:

        $ pip3 install -t python/ pytz

7. Next, we would remove the “*.dist-info” files from our package directory to reduce the size of the resulting layer.

        $ cd python
        $ sudo rm -rf *.dist-info

8. This will install all the required libraries that we need to run pandas and numpy.

9. Zip the current "python" directory and upload it to your S3 bucket. Ensure that the libraries are present in the hierarchy as given here.

        $ cd ..
        $ zip -r lambda-layer.zip python/
        $ aws s3 cp lambda-layer.zip s3://YOURBUCKETNAME

10. The "lambda-layer.zip" file can then be used to create a new layer from the Lambda console.

Answer 3

Base on aws lamda layer doc, https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html your zip package for the layer must have this structure.

my_layer.zip
  | python/numpy
  | python/numpy-***.dist-info

So what you have to do is create a folder python, and put the content of site-packages inside it, then zip up that python folder. I tried this out with a simple package and it seem to work fine.

Also keep in mind, some package require c/c++ compilation, and for that to work you must install and package on a machine with similar architecture to lambda. Usually you would need to do this on an EC2 where you install and package where it have similar architecture to the lambda.