CODEWITHSUNDEEP
amazon-web-servicesapiamazon-s3lambdamultipart
Rshad Zhran
Rshad Zhran

Reputation: 636

Upload/Download LARGE files to/from Lambda function using API Gateway without making any use of S3 Bucket

I'm implementing a serverless API, using:

  1. API Gateway
  2. Lambda
  3. S3 Bucket "If needed"

My flow is to :

  1. Call POST or PUT method with a binary file "zip", upload it to Lambda.
  2. In Lambda: unzip the file.
  3. In Lambda: Run a determined script over the extracted files.
  4. In Lambda: Generate a new zip.
  5. Return it to my desktop.

This flow is already implemented and it's working well with small files, 10MB for uploading and 6MB for downloading. But I'm getting issues when dealing with large files as it'll be the case on many occasions. To solve such issue I'm thinking about the following flow:

  1. Target file gets uploaded S3 Bucket.
  2. A new event is generated and Lambda gets triggered.

  3. Lambda Internal Tasks:

    3.1 Lambda Download the file from S3 bucket.

    3.2 Lambda Generate the corresponding WPK Package.

    3.3 Lambda Upload the generated WPK package into S3.

    3.4 Lambda returns a signed URL related to the uploaded file as a response.

But my problem with such design is that it requires more than a request to get completed. I want to do all this process in only 1 request, passing the target zip file in it and get the new zip as the response. Any Ideas, please?

My Components and Flow Diagram would be:

Component and Flow Diagram

Upvotes: 1

Views: 4857

Answers (1)

Deep Dhillon
Deep Dhillon

Reputation: 345

There are a couple of things you could do if you'd like to unzip large files while keeping a serverless approach :

  1. Use Node.js for streaming of the zip file, unzipping the file in a pipe, putting the content in a write stream pipe back to S3.
  2. Deploy your code to an AWS Glue Job.
  3. Upload the file to S3,AWS Lambda gets triggered pass the file name as the key to the glue job and the rest will be done. This way you have a serverless approach and a code that does not cause memory issues while unzipping large files

Upvotes: 3

Related Questions