CODEWITHSUNDEEP
pythondjangoamazon-web-servicesaws-lambdafacebook-prophet
Joey Coder
Joey Coder

Reputation: 3489

Bring machine learning to live production with AWS Lambda Function

I am currently working on implementing Facebook Prophet for a live production environment. I haven't done that before so I wanted to present you my plan here and hope you can give me some feedback whether that's an okay solution or if you have any suggestions.

  1. Within Django, I create a .csv export of relevant data which I need for my predictions. These .csv exports will be uploaded to an AWS S3 bucket.

  2. From there I can access this S3 bucket with an AWS Lambda Function where the "heavy" calculations are happening.

  3. Once done, I take the forecasts from 2. and save them again in a forcast.csv export

  4. Now my Django application can access the forecast.csv on S3 and get the respective predictions.

I am especially curious if AWS Lambda Function is the right tool in that case. Exports could probably also saved in DynamoDB (?), but I try to keep my v1 simple, therefore .csv. There is still some effort to install the right layers/packages for AWS Lambda. So I want to make sure I am walking in the right direction before diving deep in its documentation.

Upvotes: 1

Views: 246

Answers (1)

Abhishek Garg
Abhishek Garg

Reputation: 2288

I am a little concerned about using AWS Lambda for the "heavy" calculations. There are a few reasons.

  1. Binary size Limit: AWS Lambda has a size limit of 250MB for binaries. This is the biggest limitation we faced as you will not be able to include all the libraries like numpy, pandas, matplotlib, etc in that binary.
  2. Disk size limit: AWS only provides max disk size of 500MB for lambda execution, it can become a problem if you want to save intermediate results in the disk.
  3. The cost can skyrocket: If your lambda is going to run for a long time instead of multiple small invocations, you will end up paying a lot of money. In that case, I think you will be better off with something like EC2 and ECS.

You can evaluate linking the S3 bucket to an SQS queue and a process running on EC2 machine which is listening to the queue and performing all the calculations.

AWS Lambda Limits.

Upvotes: 2

Related Questions