Using Shapely on AWS Lambda with Python 3

Question

I am trying to set up a skill using Shapely on Lambda. I got the error

module initialization error: Could not find lib geos_c or load any of its variants ['libgeos_c.so.1', 'libgeos_c.so'].

There's a similar question for Python 2.7. I can't use the lambda-packs by ryfeus cause I'm on Python 3.6, but I figured the EC2 approach described by Graeme should work.

So I started up an EC2 instance using the Public Amazon Linux AMI version from the AWS docs

I then ran these commands

$ sudo yum -y update
$ sudo yum -y install python36 python36-virtualenv python36-pip
$ mkdir ~/forlambda
$ cd ~/forlambda
$ virtualenv -p python3 venv
$ source venv/bin/activate

and then installed Shapely and a few other packages I needed.

$ sudo yum -y groupinstall "Development Tools"
$ pip install python-dateutil
$ pip install shapely 
$ pip install pyproj 
$ pip install pyshp

I then ran my skill (on the EC2 instance), and it works! So then I copied the files at venv/lib/python3.6/site-packages, plus the myskill.py and zipped them up, uploaded to Lambda, and still get the geos_c error as shown above :(

I have been able to upload a scaled-down version of my skill (minus Shapely, but including other packages that don't come with Lambda) and it works on Lambda, so I don't think it's an error on how I am zipping or uploading.

Am I missing something? Does it make a difference that the Development Tools were installed using "sudo yum install" instead of "pip install"?

Answer 1

For some reason, the pip install of Shapely and Pyproj didn't end up in the virtualenv site-packages. From a fresh EC2 instance, I ran these commands:

$ sudo yum -y update
$ sudo yum -y install python36 python36-virtualenv python36-pip
$ mkdir ~/forlambda
$ cd ~/forlambda
$ virtualenv -p python3 venv
$ source venv/bin/activate

(venv) $ sudo yum -y groupinstall "Development Tools"
(venv) $ pip install python-dateutil
(venv) $ pip install shapely -t ~/forlambda/venv/lib/python3.6/site-packages/
(venv) $ pip install pyproj -t ~/forlambda/venv/lib/python3.6/site-packages/
(venv) $ pip install pyshp

and then zipped up all the contents of site-packages/ plus myskill.py, uploaded to Lambda, and it worked.