CODEWITHSUNDEEP
aws-api-gatewaywebhooksamazon-sqsmailgun
InTooDeep
InTooDeep

Reputation: 588

Pushing MailGun webhook onto AWS SQS

I need to process MailGun webhooks. I did implement a solution directly on our web servers to process the webhooks, but MailGun generates so many calls from a large campaign that it effectively becomes a DOS attack.

One solution I've been looking at is using AWS API Gateway to a Lambda function to then push onto an SQS queue. We can then poll the queue at a rate we can manage. Unfortunately we can't get this to work as AWS API Gateway does not support multipart/form-data content types (which some of the webhooks are). This means that our SQS messages are not well formatted / structured. The best we can do is use the $util.escapeJavaScript($input.body) function in the mapping template to create an SQS message that contains the raw string of the webhook content (with escaped javascript chars) that is effectively unparsable i.e. we can't get data out of it.

I've had a go at using Zapier to process the webhook and push directly on the SQS queue. This can parse the various content types effectively and create a nicely structured message for us, but the cost of the service is not viable.

Has anybody managed this problem in another way? Are there solutions to API Gateway not parsing the content properly? I've deliberately stayed away from MailGuns event polling API as it involves significant delays before the polled data can be 'trusted' (according to MailGun).

Basically, is there another way of getting a nicely parsed message from content types multipart/form-data and application/x-www-form-urlencoded onto the queue?

Any ideas would be much appreciated!

To add, this link higlights issues with APS Gateway and multipart\form-data content: API Gateway - Post multipart\form-data

Upvotes: 1

Views: 921

Answers (3)

Greg Fennell
Greg Fennell

Reputation: 176

Not sure if you ever came to a solution, but I have this working with the following settings.

  1. Setup your API Gateway method to use "Use Lambda Proxy integration"
  2. In your lambda (I use node.js) use busboy to work through the multi-part submission from the mailgun webhook. (use this post for help with busboy Busboy help)
  3. Make sure that any code you are going to execute after all busboy is complete is executed in the 'finish' portion of the busboy code.

Upvotes: 0

kehers
kehers

Reputation: 4160

I had the same challenge when building Suet. I ended up switching to Google Cloud functions which I really recommend. Don't waste time on Amazon API Gateway. Use Google Cloud Functions and use a middleware like multer. (You can see the source of Suet's webhook handler here).

Upvotes: 1

Abhigna Nagaraja
Abhigna Nagaraja

Reputation: 1894

As you've mentioned you can base64 encode in api gateway and call base64decode in the lambda function to retrieve the original payload (There are standard libraries in every language).

Also, note you can that you can use multipart form data for non file bodies. Get non file body from multipart/form-data using AWS API Gateway and Lambda

Upvotes: 1

Related Questions