CODEWITHSUNDEEP
amazon-web-servicesnumpyscipyscikit-learnaws-lambda
2ank3th
2ank3th

Reputation: 3109

Sklearn on aws lambda

I want to use sklearn on AWS lambda. sklearn has dependencies on scipy(173MB) and numpy(75MB). The combined size of all these packages exceeds AWS Lambda disk space limit of 256 MB.

How can I use AWS lambda to use sklearn?

Upvotes: 9

Views: 7281

Answers (3)

elier
elier

Reputation: 459

I wanted to do the same, and it was very difficult indeed. I ended up buying this layer that includes scikit-learn, pandas, numpy and scipy.

https://www.awslambdas.com/layers/3/aws-lambda-scikit-learn-numpy-scipy-python38-layer

There is another layer that includes xgboost as well.

Upvotes: 0

Entropic Thunder
Entropic Thunder

Reputation: 176

This guy gets it down to 40MB, though I have not tried it myself yet.

The relevant Github repo.

Upvotes: 6

user6502956
user6502956

Reputation:

there is a two way to do this
1) installing the modules dynamically
2) aws batch

1) installing the modules dynamically 

def lambdahandler():
   #install numpy package 
   # numpy code 
   #uninstall numpy package
   ## now install Scipy package 
      # execute scipy code

or vice versa depends on your code

2) using Aws batch
This is the best way where you don't have any limitation regarding Memory space. just you need to build a Docker image and need to write an all required packages and libraries inside the requirement.txt file.

Upvotes: 0

Related Questions