module initialization error: Cannot load native module 'Crypto.Cipher._raw_ecb' at AWS lambda

Question

I am making service using AWS lambda. I am using PyCryptodome for encryption and decryption. I am able to test my application locally, but when I upload to AWS lambda for decrypting. I get the error as

module initialization error: Cannot load native module 'Crypto.Cipher._raw_ecb': Trying '_raw_ecb.cpython-36m-x86_64-linux-gnu.so': /var/task/Cryptodome/Util/../Cipher/_raw_ecb.cpython-36m-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory, Trying '_raw_ecb.abi3.so': /var/task/Cryptodome/Util/../Cipher/_raw_ecb.abi3.so: cannot open shared object file: No such file or directory, Trying '_raw_ecb.so': /var/task/Cryptodome/Util/../Cipher/_raw_ecb.so: invalid ELF header

My code for decryption is

    def blowfish_decrypt(enc):
        secret_key = b"somestring"
        iv = b"somerandomiv"
        logger.info("in the decrypter")
        crypt_obj = bf_cbc.new(secret_key, bf_cbc.MODE_CBC, IV=iv)
        original = crypt_obj.decrypt(base64.b64decode(enc))
        original = original.decode("utf-8")
        logger.info("decrypted")
        return original

I was following the resource: https://github.com/pyinstaller/pyinstaller/issues/2125, but this didn't help me either.

after applying all the details as specified I am getting the same above error.

Answer 1

We proceed with Docker and do something very similar without layers, which will be in the next iteration moving forward:

In the directory of interest in the command line tool: touch Dockerfile followed by sudo nano Dockerfile.

Copy and paste the following into the Dockerfile but with the other libraries added accordingly (see below)

Dockerfile:

# Use Python 3.12 as the base image to avoid building it from source
FROM python:3.12-slim

# Install required utilities
RUN apt-get update && \
    apt-get install -y gcc zip && \
    rm -rf /var/lib/apt/lists/*

# Create function directory and set working directory
WORKDIR /function

# Copy your Lambda function code
COPY lambda_function.py /function/lambda_function.py

# Install dependencies into the function directory
RUN pip3 install -t /function requests #other libraries here

# Install pydantic and pycryptodome specifically for Python 3.12 with manylinux2014 compatibility
RUN pip3 install \
    --platform manylinux2014_x86_64 \
    --target=/function \
    --implementation cp \
    --python-version 3.12 \
    --only-binary=:all: \
    --upgrade pydantic pycryptodome

# Package everything into a zip file for Lambda
RUN zip -r /function/linux-lambda.zip .

Then we ran the following in bash/terminal:

docker build -t genzipfile .

docker run -d --name genzipfile_container genzipfile

docker cp genzipfile_container:/function/linux-lambda.zip {your_folder_path}

docker rm -f genzipfile_container

Navigate to the Lambda console and import the linux-lambda.zip file then the process is finished.

Answer 2

Expanding on @Milan's answer, The M1 Mac's do not seem to support the Python3.6 image anymore (I'm sure there is a way to get this to work, but didn't for me). Considering that Python3.6 has been depreciated you can do the following to run it on a newer version of Python.

docker pull shogo82148/lambda-python:build-3.11-arm64

Then run

docker run --rm -v `pwd`:/var/task shogo82148/lambda-python:build-3.11-arm64 pip install pycryptodome -t pycryptodome

And then deploy to Lambda. Works a treat.

Answer 3

You need x86 based CPU and Linux environment to run pip install. The binary from Apple silicon (M1/M2/...) will pass the self test but won't work on Lambda.

On AWS, use x86 based EC2 or CodeBuild.

This is not the issue on original question, but I am sure some people will encounter same issue as me now.

Answer 4

if you are using conda, enter these lines in cmd:

conda activate your_env_name
conda install pycrypto

If you are using pip, upload the module via pip. After installed, try again to see if you are getting the same error.

Answer 5

This is happening because the pycryptodome module installed on your local machine is not compatible in lambda. so there are two ways we can fix this.

1. Use docker to pull amazonlinux image and install pycryptodome using pip install. then export the pycryptodome module to lambda layers.
2. Start a ec2 instance, must be amazonlinux and install pycryptodome in it. then either download the module using winscp or cli to your local. Create a lambda layer package using the downloaded module and upload it to lambda layer.

make sure to follow below guidline for creating lambda layer package. Import libraries in lambda layers

Answer 6

I installed and run cryptodome 3.9.4 on AWS Lambda successfully (only Python 3.6).

I put the package in github. It is required to put it into your microservice.

https://github.com/grmagalhaes/python-utilities/tree/master/Crypto

Answer 7

It looks like your local dev environment is not compatible with the Lambda execution environment. The native libraries that PyCryptodome uses are not portable across these two environments; it matters in which env the library was pip installed.

One way to fix it is to use Lambci docker image to build the library and then add it to the zip file. Assuming you have Docker installed, do

docker pull lambci/lambda:build-python3.6
docker run --rm -v `pwd`:/var/task lambci/lambda:build-python3.6 pip install pycryptodome -t pycryptodome

This will pip install the lib in the docker environment. After the command finishes, you'll have it available in the pycryptodome local dir.

For a more automated/repeatable way, have a look at AWS SAM and aws-sam-cli which gives you some very useful commands to build, package and deploy your Lambda apps.