XGBoostError: XGBoost Library (libxgboost.so) could not be loaded

Question

Full Error:

[ERROR] XGBoostError: XGBoost Library (libxgboost.so) could not be loaded.
Likely causes:
  * OpenMP runtime is not installed (vcomp140.dll or libgomp-1.dll for Windows, libgomp.so for UNIX-like OSes)
  * You are running 32-bit Python on a 64-bit OS
Error message(s): ['libgomp.so.1: cannot open shared object file: No such file or directory']

Infrastructure:

• AWS Lambda Function

• I am importing the XGBoost in the Lambda function and I get above error when I Try to run it.

• This error has started when I upgraded the Lambda Function Runtime version from Python 3.6 to Python 3.9

• I am importing the XGBoost from the zip I have kept in the S3 bucket and Lambda has a code to import it.

• I also tried packaging the libgomp.so.1 with the xgboost zip but that also did not worked.

Answer 1

use the correct amazon linux base, aws python 3.9 runtime is based on Amazon linux 2 so you need to build and package dependencies in the same environment

run the following to start with the correct environment

docker run -it --rm amazonlinux:2

when you are inside the docker container run this

yum update -y
yum install -y gcc gcc-c++ make python3 python3-devel
python3 -m venv /app/venv
source /app/venv/bin/activate
pip install --upgrade pip
pip install xgboost

now do this

find / -name "libgomp.so.1"

Copy this library to your Lambda deployment package

cp /usr/lib64/libgomp.so.1 /app/venv/lib/python3.9/site-packages/

now zip the deployment package

cd /app/venv/lib/python3.9/site-packages/
zip -r /tmp/xgboost_package.zip .

now upload to S3 and deploy

I tried producing your error, but yeah these things that i mentioned above are fixed

Answer 2

try using an amazon linux compatible prebuilt wheel

you can using this container as it has same env as lambda

docker run -it --rm amazonlinux:2

in docker container install requirements make sure to use venv

yum install -y gcc gcc-c++ python3 python3-devel