CODEWITHSUNDEEP
amazon-web-servicesapi-gatewayaws-api-gateway
Akshit Gupta
Akshit Gupta

Reputation: 77

Large Payload (Text Files) in request and response of the API of API Gateway

I have a use case where my API shall be exposed to the external clients via the API Gateway. The request params include a large text file (~100MB) and some other parameters in the payload. After doing some processing on the data from the file, I intend to send the updated file in the response body.

Please help out as how can I solve this use case of mine. Thanks

Challenges:

The processing of the API request could exceed the API Gateway timeout limit of 30 seconds. The payload of the request and the response exceeds the payload limit of 10 MB.

Upvotes: 2

Views: 825

Answers (1)

maafk
maafk

Reputation: 6876

You'll most likely want this to be an asynchronous process, where you:

  • Create a pre-signed post urls where users can upload their 100MB text files directly to S3
    • Optionally notify user saying "Thanks! You'll be notified when your file is ready"
  • Have that S3 event trigger a lambda function to process the data in the file, save to another location in S3
  • Generate a Pre signed url on the transformed text file
  • Present that url to the user for download

Uploading, transforming and downloading a ~100MB text file is probably not a synchronous process you'd want users waiting around for.

Upvotes: 2

Related Questions