CODEWITHSUNDEEP
aws-lambdaamazon-efs
Ighenvex
Ighenvex

Reputation: 13

Importing LightGBM into AWS Lambda

I currently have an AWS Lambda function that's connected to an EFS. The EFS is storing all of my dependencies, such as numpy, pandas, and lightgbm. I have no trouble importing any package except lightgbm. When I do, it raises the error:

"libgomp.so.1: cannot open shared object file: No such file or directory"

How do I fix it?

I've tried uploading LightGBM as a lambda layer, as well as packaging libgomp into the lambda layer, but neither have worked.

Upvotes: 1

Views: 575

Answers (2)

Dinh Jylee
Dinh Jylee

Reputation: 41

I am using Lambda ARM64, Python3.12 and this is the way I deal with this issue:

  • Use LightGBM <=3.3.3 because glibc 2.26 (Amazon Linux 2) will not match for the requirement of newer LightGBM (ref)
  • Pull and run base ARM64 image of AWS: docker pull public.ecr.aws/sam/build-python3.12:latest-arm64
  • Access to Terminal of the above container and install libgomp dnf install -y libgomp
  • Copy libgomp to local machine
docker cp base-lambda-python3.12-arm64:/lib64/libgomp.so.1 .
docker cp base-lambda-python3.12-arm64:/lib64/libgomp.so.1.0.0 .
  • Copy them to built aws-sam:
cp ./libgomp.so.1 ./libgomp.so.1.0.0 .aws-sam/build/YourAlphaFunction
  • Add environment to CloudFormation SAM Template File:
Environment:
    Variables:
      LD_LIBRARY_PATH: "/var/task:/usr/lib64:/lib"

I hope it helps someone!

Upvotes: 0

nuri
nuri

Reputation: 240

After much effort and numerous attempts over the course of two days, I was able to resolve the issue. It's worth noting that the versions of libgomp.so and glibc can vary depending on the Amazon Lambda runtime version. For more information, refer to https://repost.aws/questions/QUrXOioL46RcCnFGyELJWKLw/glibc-2-27-on-amazon-linux-2. enter image description here To address this, you can create a custom AWS Lambda layer by provisioning an Amazon Linux 2 instance on an EC2, downloading the required libgomp.so and glibc packages, and packaging them into a Lambda layer. Ensure that the AWS Lambda layer objects are located under the /lib folder, as specified in the documentation: https://docs.aws.amazon.com/lambda/latest/dg/packaging-layers.html.

Once this layer is added, the Lightgbm library should be importable in your Lambda environment.

here my layer for python 3.8! AWS-Lambda-Layers-v0.zip

Upvotes: 0

Related Questions