CODEWITHSUNDEEP
pythonamazon-web-servicesaws-lambdaaws-lambda-layers
Ericson Willians
Ericson Willians

Reputation: 7845

Is it possible to add paths to the PATH environment variable through serverless.yml?

When I create an AWS Lambda Layer, all the contents / modules of my zip file go to /opt/ when the AWS Lambda executes. This easily becomes cumbersome and frustrating because I have to use absolute imports on all my lambdas. Example:

import json
import os
import importlib.util
spec = importlib.util.spec_from_file_location("dynamodb_layer.customer", "/opt/dynamodb_layer/customer.py")
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)

def fetch(event, context):

    CustomerManager = module.CustomerManager
    customer_manager = CustomerManager()

    body = customer_manager.list_customers(event["queryStringParameters"]["acquirer"])

    response = {
        "statusCode": 200,
        "headers": {
            "Access-Control-Allow-Origin": "*"
        },
        "body": json.dumps(body)
    }

    return response

So I was wondering, is it possible to add these /opt/paths to the PATH environment variable by beforehand through serverless.yml? In that way, I could just from dynamodb_layer.customer import CustomerManager, instead of that freakish ugliness.

Upvotes: 5

Views: 3462

Answers (4)

MikeW
MikeW

Reputation: 6120

Setting of PYTHONPATH variable is not required, as long as you place items correctly inside the zip file.

Simple modules, and package directories, these should be placed inside a directory "python", and then the whole python/ placed into the zip file for uploading to AWS as a layer. Don't forget to add the "compatible runtimes" (eg Python 3.6, 3.7, 3.8 ...) settings for the layers.

So as an example:

python/
- my_module.py
- my_package_dir
-- __init__.py
-- package_mod_1.py
-- package_mod_2.py

which then get included in the zip file.

zip -r my_layer_zip.zip python/

The modules can then be imported without any more ado, when accessed as a layer:

....
import my_module
from my_package.package_mod_2 import mod_2_function
....

You can see the package structure from within the lambda if you look at '/opt/python/' which will show my_module.py, my_package/ etc., this is easily tested using the AWS Lambda test function, assuming the layer is attached to the function (or else the code will error)

import json
import os

def lambda_handler(event, context):
    # TODO implement

    dir_list = os.listdir('/opt/python/')

    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!'),
        'event': json.dumps(event),
        '/opt/python/': dir_list
    }

Upvotes: 0

Milan Cermak
Milan Cermak

Reputation: 8064

In the zip archive, the module needs to be placed the in a python subdirectory so that when it is extracted as a layer in Lambda, it is located in /opt/python. That way you'll be able to directly import your module without the need for importlib.

It's documented here or see this detailed blogpost from an AWS dev evangelist for more.

Upvotes: 0

A.Khan
A.Khan

Reputation: 4002

I've Lambda layer for Python3.6 runtime. My my_package.zip structure is:

my_package.zip
 - python
   - lib
     - python3.6
       - site-packages
         - customer

All dependencies are in build folder in project root: e.g. build/python/lib/python3.6/site-packages/customer

Relevant section of my serverless.yml

layers:
  my_package:
    path: build             
    compatibleRuntimes:     
      - python3.6

In my Lambda I import my package like I would do any other package: import customer

Upvotes: 1

eagle33322
eagle33322

Reputation: 227

Have you tried setting your PYTHONPATH env var? https://stackoverflow.com/a/5944201/6529424

Have you tried adding to sys.path? https://stackoverflow.com/a/12257807/6529424

Upvotes: 0

Related Questions