CODEWITHSUNDEEP
pythonaws-lambda
Justin
Justin

Reputation: 4843

Can't import `yaml` module from AWS Lambda layer

I have a very simple Python3.6 Lambda function, the only complexity being that I want to include pyyaml within it. index.py looks like this -

import yaml

def handler(event, context):
    return event

Everything works fine if I bundle pyyaml within my Lambda deployable, like this -

(aws_layer_demo) justin@justin-XPS-13-9360:~/work/aws_demos/aws_layer_demo$ jar -tvf  tmp/layer-demo-dev/2019-05-31-06-27-41.zip 
    59 Fri May 31 06:27:13 BST 2019 index.py
     0 Fri May 31 06:19:32 BST 2019 __init__.py
  4881 Fri May 31 06:15:37 BST 2019 yaml/composer.py
 25554 Fri May 31 06:15:37 BST 2019 yaml/constructor.py
  3294 Fri May 31 06:15:37 BST 2019 yaml/cyaml.py
                                    {... etc ...}

I can ping the Lambda function and get a response, no problem.

Now I want to test including pyyaml within a layer. So I now have two deployables, one for the app -

(aws_layer_demo) justin@justin-XPS-13-9360:~/work/aws_demos/aws_layer_demo$ jar -tvf  tmp/layer-demo-dev/2019-05-31-06-43-42.zip 
    59 Fri May 31 06:27:13 BST 2019 index.py
     0 Fri May 31 06:19:32 BST 2019 __init__.py

and one for the layer -

(aws_layer_demo) justin@justin-XPS-13-9360:~/work/aws_demos/aws_layer_demo$ jar -tvf tmp/layer-demo-layer-dev/2019-05-31-06-43-42.zip 
     0 Fri May 31 06:43:42 BST 2019 __init__.py
  4881 Fri May 31 06:15:37 BST 2019 yaml/composer.py
 25554 Fri May 31 06:15:37 BST 2019 yaml/constructor.py
  3294 Fri May 31 06:15:37 BST 2019 yaml/cyaml.py
                                    {... etc ...}

I have deployed both using a Cloudformation stack (via S3) and it looks like both the function and layer have been deployed correctly -

(aws_layer_demo) justin@justin-XPS-13-9360:~/work/aws_demos/aws_layer_demo$ ./scripts/show_function_config.sh dev
------------------------------------------------------------------------------------------------
|                                   GetFunctionConfiguration                                   |
+--------------+-------------------------------------------------------------------------------+
|  CodeSha256  |  MImn7X/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=                                 |
|  CodeSize    |  372                                                                          |
|  Description |                                                                               |
|  FunctionArn |  arn:aws:lambda:eu-west-1:XXXXXXXXXXXX:function:layer-demo-dev                |
|  FunctionName|  layer-demo-dev                                                               |
|  Handler     |  index.handler                                                                |
|  LastModified|  2019-05-31T05:44:14.089+0000                                                 |
|  MemorySize  |  512                                                                          |
|  RevisionId  |  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX                                         |
|  Role        |  arn:aws:iam::XXXXXXXXXXXX:role/layer-demo-dev-AppFunctionRole-XXXXXXXXXXXX   |
|  Runtime     |  python3.6                                                                    |
|  Timeout     |  5                                                                            |
|  Version     |  $LATEST                                                                      |
+--------------+-------------------------------------------------------------------------------+
||                                           Layers                                           ||
|+-------------+------------------------------------------------------------------------------+|
||  Arn        |  arn:aws:lambda:eu-west-1:XXXXXXXXXXXX:layer:layer-demo-dev:2                ||
||  CodeSize   |  41184                                                                       ||
|+-------------+------------------------------------------------------------------------------+|
||                                        TracingConfig                                       ||
|+-------------------------------+------------------------------------------------------------+|
||  Mode                         |  PassThrough                                               ||
|+-------------------------------+------------------------------------------------------------+|
||                                          VpcConfig                                         ||
|+---------------------------------------------------------------+----------------------------+|
||  VpcId                                                        |                            ||
|+---------------------------------------------------------------+----------------------------+|

In particular note "Layers" in the above, and the fact that the CodeSize > 0 (ie I haven't accidentally deployed a size zero zipfile)

But when I ping my newly layerized Lambda function, I now get -

Unable to import module 'index'

And if I delve a bit deeper into the Cloudwatch logs I get -

Unable to import module 'index': No module named 'yaml'

So. It feels like I have probably somehow mis-structured my layer package (in terms of directory structue) so that Lambda doesn't see yaml on internal AWS PYTHONPATH. Or something. Can anyone advise me where I might be going wrong ?

Many thanks.

Upvotes: 5

Views: 7221

Answers (1)

Justin
Justin

Reputation: 4843

Turns out this was due to a couple of factors.

First is that you have to be very careful about how you structure the layer package structure. It's not enough just to zip the dependencies - in the case of Python Lambda functions they have to be installed in a python directory within the zip file. That directory changes depending on the runtime you are using. Not particularly clear from the Lambda docs, but more details here -

https://medium.com/@adhorn/getting-started-with-aws-lambda-layers-for-python-6e10b1f9a5d

Second is that when deploying the layer you need to specify the compatible runtime(s) - so if you're using the AWS CLI it's something like -

aws lambda publish-layer-version --layer-name $appname --content S3Bucket=$s3bucketname,S3Key=$appname/layer.zip --compatible-runtimes python3.7

Follow these two steps and your layer should work fine.

Upvotes: 3

Related Questions