CODEWITHSUNDEEP
amazon-web-servicesaws-api-gatewayamazon-kinesis-firehose
credizian
credizian

Reputation: 479

What should the mapping template look like for AWS Firehose PutRecordBatch in API Gateway?

I've successfully setup an API that has Kinesis Firehose integration with AWS API Gateway using PutRecord using these instructions (https://aws.mannem.me/?p=1152 - note: it says insecure but I still clicked through since I needed it).

I'm trying to setup an API for PutRecordBatch (essentially allows for more than one record to be written at a time) but I keep getting

{ "__type": "SerializationException" }

Based on hours of research, API gateway throws that error when incoming API call format doesn't match the mapping template noted in the Integration Request. I'm struggling to figure out how to fix my mapping template.

Here's my mapping template:

{
    "StreamName": "$input.path('DeliveryStreamName')",
    "Records": [
           #foreach($elem in $input.path('$.Records'))
              {
                "Data": "$util.base64Encode($elem.Data)",
              }#if($foreach.hasNext),#end
            #end
        ]
}

Here's the test data that I'm sending:

{
    "DeliveryStreamName": "test",
    "Records": [{
        "Data": "SampleDataStringToFirehose"
    },
    {
        "Data": "SampleDataStringToFirehose2"
    }]
}

Upvotes: 2

Views: 1242

Answers (2)

Luis
Luis

Reputation: 716

Your example helped me a lot so I wanted to complement it, just in case anybody else run into my specific scenario.

In my case, instead of a simple string I needed to send a JSON object, similar to this:

{
   "DeliveryStreamName": "test",
   "Records": [{
       "Data": {"foo": "bar", "count": 321}
    },
    {
       "Data": {"foo1": "bar1", "count": 10}
    }]
}

In this case, what happened when I used the template in your example is that the object is stored in a non JSON format, which is not suitable for further analysis.

With a simple adjustment to the template, you could store a correctly formated JSON object:

{
    "StreamName": "$input.path('DeliveryStreamName')",
    "Records": [
           #foreach($elem in $input.path('$.Records'))
              {
                #set($jsonPath = "$.Records[$foreach.index].Data")
                "Data": "$util.base64Encode($input.json($jsonPath))"
              }#if($foreach.hasNext),#end
            #end
     ]
}

Upvotes: 0

credizian
credizian

Reputation: 479

So dumb but the mapping template has an error: there is an extra comma in there at the end of

"Data": "$util.base64Encode($elem.Data)",

that's causing the issue. Below is the correct version:

{
    "DeliveryStreamName": "$input.path('$.DeliveryStreamName')",
    "Records": [
           #foreach($elem in $input.path('$.Records'))
              {
                "Data": "$util.base64Encode($elem.Data)"
              }#if($foreach.hasNext),#end
            #end
        ]
}

Upvotes: 2

Related Questions