Getting PIL/Pillow 4.2.1 to upload properly to AWS Lambda Py3.6

Question

Background

I have been struggling for the past few days to deploy a Lambda that uses Pillow, and I am deploying using Python 3.6. It may be noteworthy also that I am developing this on a Windows 10 environment.

First Attempts

I began by having pip install my packages strictly in my workspace by doing the following:

pip3 install pillow -t "D:\Work and Projects\...\...\<projectdir>\pillow"

I have other packages, and tried installing the packages in the same manor, one of them specifically was praw and I did so by:

pip3 install praw -t "D:\Work and Projects\...\...\<projectdir>\praw"

After zipping the contents of my project together, I uploaded my package up to Lambda and upon my first test I received the error:

Unable to import module 'motw_lambda': cannot import name '_imaging'

I then removed the Pillow package in an attempt to see where this issue was stemming from (Pillow or praw or one of the other packages). With Pillow removed, the execution succeeded. I then removed the pillow package in my package and tried:

pip3 install pillow -t "D:\Work and Projects\...\...\<projectdir>\PIL"

and

pip3 install pillow -t "D:\Work and Projects\...\...\<projectdir>\Pillow"

But got the same error with the package '_imaging'.

Further Attempts

I then followed the directions per this resource and this. I also tried using virualenv and even lambda-uploader.

Strange enough, I get the same error! I am all out of options here, and feel either I am doing something dumb, or that this is not possible on Lambda-Python3.6 currently (Although I can't image someone else hasn't used pillow in a py3.6-lambda yet...)

Any info, help, or generic resources would be appreciated!

Answer 1

Finally found a pretty easy solution. The trick is to install Pillow in a directory locally, zip it up, and then create a Lambda Layer. Now, the Python version you run locally must match the Python version of the Lambda. In my case locally I have Python 3.10, and the Lambda is 3.9, so I installed 3.9 locally just to use for this.

On Mac I used homebrew to install pyenv to get 3.9, but however you want to install is fine.

    !Only needed if you do not have the correct version of Python locally!
        brew install pyenv
        pyenv install 3.9 (or whatever version)
(if you do this, then you can run 'pyenv local 3.9.16' or whatever version you installed, then 'Python3 --versions' should show that version.

For example, pyenv installed in MY_USER_DIR/.pyenv/versions/3.9.16/bin/python3 OR you can set your env to with pyenv to use the specific version.

Then cd into an empty dir, and run the following:

YOUR_LOCAL_PATH_TO_CORRECT_VERSION/python3 -m pip install \
    --platform manylinux2014_aarch64 \
    --target=./python/lib/python3.9/site-packages \
    --implementation cp \
    --python 3.9 \
    --only-binary=:all: --upgrade \
    Pillow

In the above, manylinux2014_aarch64 is for the ARM runtime. So your Lambda will have to be set to that, or find the value for the X86 version. And if you are not using 3.9, then change that to your version.

Your are now good to go to create the zip for your layer. Just run the following, and then upload the zip to a layer, and make sure you select the correct Python version and architecture for your Lambda.

in the same dir you ran the above command:
zip -r ../pillowLayer.zip .

Answer 2

Basically, you have to compile the libraries (eg, PIL) either using Docker or, even better, an EC2 instance.

1. Launch an Docker container like this: docker run --rm -it -v "%cd%:/code" lambci/lambda:build-python3.6 sh

2. Inside there, navigate to the /code dir (cd /code), create a virtualenv (virtualenv env), activate it (source env/bin/activate) and finally, install your library (pip install pillow).

3. Once you have installed your library, you can exit the container. The secret here is to move your package library to the root folder (where your main .py file is). For example, move the folder env/lib/python3.6/site-packages/PIL to the root.

Then, zip your PIL folder together with your .py file and you're set!

Full Example:

The following example compiles and compresses PIL and other common Python libraries to run in AWS Lambda.

Dockerfile:

FROM lambci/lambda:build-python3.6
WORKDIR /code
CMD ["sh", "entrypoint.sh"]

entrypoint.sh:

#!/bin/sh

set -ex

cd /code/

if [ ! -d "PIL" ]; then
    # Create virtual env, activate it and install PIL
    virtualenv env && source env/bin/activate && pip install pillow requests

    # Copy necessary files to the root folder
    rm -f build-output.zip
    #mkdir PIL
    cp -f -r env/lib/python3.6/site-packages/PIL .
    cp -f -r env/lib/python3.6/site-packages/requests .
    
    # below are the dependencies for the requests pkg
    cp -f -r env/lib/python3.6/site-packages/urllib3 .
    cp -f -r env/lib/python3.6/site-packages/certifi .
    cp -f -r env/lib/python3.6/site-packages/chardet .
    cp -f -r env/lib/python3.6/site-packages/idna .
    
    # Remove temp files
    rm -r env
fi

# ZIP it
zip -9 build-output *.py 
zip -9 -r build-output PIL
zip -9 -r build-output requests
zip -9 -r build-output urllib3
zip -9 -r build-output certifi
zip -9 -r build-output chardet
zip -9 -r build-output idna

To build (Windows):

docker build -t build_lambda .
docker run --rm -v "%cd%:/code" build_lambda

Answer 3

For anyone else also new to aws python and running into this issue, you can use the layers feature, and there are existing layers here you can link to and this worked for me.

https://github.com/keithrozario/Klayers

Pillow specifically on us-east-1:

arn:aws:lambda:us-east-1:770693421928:layer:Klayers-python38-Pillow:2

Answer 4

You can use a precompiled version of the PIL available here: https://github.com/Miserlou/lambda-packages

Just extract PIL folder to the deployment package and it should work.