CODEWITHSUNDEEP
pythonamazon-web-servicesnumpyaws-lambda
Sebastian Łącki
Sebastian Łącki

Reputation: 197

Using numpy in AWS Lambda

I'm looking for a work around to use numpy in AWS lambda. I am not using EC2 just lambda for this so if anyone has a suggestion that'd be appreciated. Currently getting the error:

cannot import name 'multiarray'

Using grunt lambda to create the zip file and upload the function code. All the modules that I use are installed into a folder called python_modules inside the root of the lambda function which includes numpy using pip install and a requirements.txt file.

Upvotes: 10

Views: 27566

Answers (7)

vikhil sallagargi
vikhil sallagargi

Reputation: 1

Title:

[Problem] Issue with dependencies while using Matplotlib and NumPy in AWS Lambda

ISSUES :

{ "errorMessage": "module 'os' has no attribute 'add_dll_directory'", "errorType": "AttributeError", "stackTrace": [ " File "/var/lang/lib/python3.8/imp.py", line 234, in load_module\n return load_source(name, filename, file)\n", " File "/var/lang/lib/python3.8/imp.py", line 171, in load_source\n module = _load(spec)\n", " File "", line 702, in _load\n", " File "", line 671, in _load_unlocked\n", " File "", line 843, in exec_module\n", " File "", line 219, in _call_with_frames_removed\n", " File "/var/task/lambda_function.py", line 3, in \n import numpy as np\n", " File "/opt/python/numpy/init.py", line 112, in \n _delvewheel_patch_1_5_2()\n", " File "/opt/python/numpy/init.py", line 109, in _delvewheel_patch_1_5_2\n os.add_dll_directory(libs_dir)\n" ] }

Description:

I encountered an issue while trying to use Matplotlib and NumPy in an AWS Lambda function. Despite specifying specific versions of dependencies, I kept encountering errors related to missing modules or incompatible versions. After troubleshooting, I found a solution that resolved the issue.

Solution:

After thorough investigation, I discovered that the problem was caused by incompatible dependencies. To resolve the issue, I downgraded certain dependencies to specific versions that are known to work well together.

Dependencies:

  • certifi==2019.6.16
  • cftime==1.0.3.4
  • chardet==3.0.4
  • cycler==0.10.0
  • idna==2.8
  • kiwisolver==1.1.0
  • matplotlib==3.1.1
  • netCDF4==1.5.1.2
  • numpy==1.17.1
  • pyparsing==2.4.2
  • python-dateutil==2.8.0
  • requests==2.22.0
  • six==1.12.0
  • urllib3==1.25.3

Tested Environment:

  • Local development environment (mac os, Python 3.8)
  • AWS Lambda

Tags:

python, matplotlib, numpy, dependencies, aws-lambda

Feedback:

I'm open to feedback on this solution or alternative approaches that others would recommend. Any suggestions or insights would be greatly appreciated.

Upvotes: -1

Saro
Saro

Reputation: 1

Add numpy layer in this way:

  • Go on your lambda function

  • select add a new layer

  • add it using this arn: arn:aws:lambda:eu-central-1:770693421928:layer:Klayers-p39-numpy:7

(change your zone if you are not in eu-central-1)

Let me know if it will work

Upvotes: 0

user_5
user_5

Reputation: 576

An easy way to make your lambda function support the numpy library for python 3.7:

  1. Go to your lambda function page
  2. Find the Layers section at the bottom of the page.
  3. Click on Add a layer.
  4. Choose AWS layers as layer source.
  5. Select AWSLambda-Python37-Scipy1x as AWS layers.
  6. Select 37 for version.
  7. And finally click on Add.

Now your lambda function is ready to support numpy.

Upvotes: 12

appleios
appleios

Reputation: 9

1.) Do a Pip install of numpy to a folder on your local machine.

2.) once complete, zip the entire folder and create a zip file.

3.) Go to AWS lambda console, create a layer and upload zip file created in step 2 there and save the layer.

4.) After you create your lambda function, click add layer and add the layer you created. That's it, import numpy will start working.

Upvotes: -1

DaveRGP
DaveRGP

Reputation: 1480

As of 2018 it's best to just use the inbuilt layers functionality.

AWS have actually released a pre-made one with numpy in it: https://aws.amazon.com/blogs/aws/new-for-aws-lambda-use-any-programming-language-and-share-common-components/

Upvotes: 1

launchpadmcquack
launchpadmcquack

Reputation: 1051

I was unable to find a good solution using serverless plugins, but I did find a good way with layers. See Serverless - Numpy - Unable to find good bind path format

Upvotes: 0

cailan s
cailan s

Reputation: 131

Updated to include the solution here, rather than a link:

After much effort, I found that I had to create my deployment package from within a python3.6 virtualenv, rather than directly from the host machine. I did the following within a Ubuntu 16.04 docker image. This assumes that you have python3.6, virtualenv and awscli already installed/configured, and that your lambda function code is in the ~/lambda_code directory:

1) cd ~ (We'll build the virtualenv in the home directory)

2) virtualenv venv --python=python3.6 (Create the virtual environment)

3) source venv/bin/activate (Activate the virtual environment)

4) pip install numpy

5) cp -r ~/venv/lib/python3.6/site-packages/* ~/lambda_code (Copy all installed packages into root level of lambda_code directory. This will include a few unnecessary files, but you can remove those yourself if needed)

6) cd ~/lambda_code

7) zip -r9 ~/package.zip . (Zip up the lambda package)

8) aws lambda update-function-code --function-name my_lambda_function --zip-file fileb://~/package.zip (Upload to AWS)

Your lambda function should now be able to import numpy with no problems.

If you want a more out-of-the-box solution, you could consider using serverless to deploy your lambda function. Before I found the above solution, I followed the guide here and was able to run numpy successfully in a python3.6 lambda function.

Upvotes: 8

Related Questions