CODEWITHSUNDEEP
amazon-web-servicesaws-lambdaamazon-iamamazon-kinesisamazon-kinesis-firehose
user2884707bond
user2884707bond

Reputation: 569

Is it possible to perform PutRecord with Kinesis FireHose on AWS Lambda using .Net Core 2.2?

The reason why I ask about this because I don't see any official documents mentioning performing PutRecord from AWS Lambda function to FireHose. I want to perform PutRecord from AWS Lambda on Kinesis FireHose. I have also given appropriate PutRecord policy to the AWS Lambda function that I am trying to PutRecord from. I get the following error when PutRecord action is performed from AWS Lambda using .Net 2.2

User: arn:aws:sts::accountnumber:assumed-role/listener-role/lambda is not authorized to perform: kinesis:PutRecord on resource: arn:aws:kinesis:us-west-1:accountnumber:assumed:stream/firehose-stream

I have a policy as follow 

{
  "permissionsBoundary": {},
  "roleName": "listener-role",
  "policies": [
    {
      "document": {
        "Version": "2012-10-17",
        "Statement": [
          {....},
          {
            "Effect": "Allow",
            "Action": [
              "firehose:PutRecord",
              "firehose:PutRecordBatch"
            ],
            "Resource": [
              "*"
            ]
          }
        ]
      },
      "name": "policy",
      "type": "inline"
    }
  ],
  "trustedEntities": [
    "lambda.amazonaws.com"
  ]
}

enter image description here

.Net Snipped for Putting record on Kinesis FireHose

_kinesisClient is AmazonKinesisClient

        MemoryStream recordStream = new MemoryStream();
        IFormatter formatter = new BinaryFormatter();
        formatter.Serialize(recordStream, data);
        var request = new PutRecordRequest
        {
            PartitionKey = Guid.NewGuid().ToString(),
            Data = recordStream,
            StreamName = Environment.GetEnvironmentVariable("KinesisStream")
        };
        await _kinesisClient.PutRecordAsync(request);

Upvotes: 0

Views: 2190

Answers (2)

Cloudkollektiv
Cloudkollektiv

Reputation: 14769

You are trying to put to data in a Kinesis Data Stream. Your policy allows you to put data in a Kinesis Firehose. This can be somewhat confusing because of the different flavours of Kinesis. If indeed you are trying to put data in a Kinesis Data Stream, you should change your policy action to kinesis:Put*.

On the other hand, if you want to put data in a Kinesis Firehose, change your .NET code to something like this (I am not a .NET expert):

var putRecordRequest = new PutRecordRequest();
var deliveryStreamName = Environment.GetEnvironmentVariable("KinesisStream");

putRecordRequest.setDeliveryStreamName(deliveryStreamName);

var record = new Record().withData(ByteBuffer.wrap(data.getBytes()));

putRecordRequest.setRecord(record);

// Put record into the DeliveryStream
firehoseClient.putRecord(putRecordRequest);

Upvotes: 3

user2884707bond
user2884707bond

Reputation: 569

I was using the wrong client to putrecord on the Kinesis Firehose. The KinesisFireHose client looks something like this.

Nuget Package: AWSSDK.KinesisFirehose" Version="3.3.103.28"

serviceCollection.AddScoped<IAmazonKinesisFirehose, AmazonKinesisFirehoseClient>();

Use the dependency injected IAmazonKinesisFirehose

var data = "{\"casenumber\": \"" + 123 + "\"}";

// convert string to stream
var byteArray = Encoding.UTF8.GetBytes(data);

var putRecordRequest = new PutRecordRequest {
 DeliveryStreamName = Environment.GetEnvironmentVariable("KinesisFirehose"), // AWS console -> Data FIrehose -> "Firehose delivery streams" 
  Record = new Record {
   Data = new MemoryStream(byteArray)
  }
};

// Put record into the DeliveryStream
Console.WriteLine($ "PutRecordAsync: {data}");

Console.WriteLine("Writing EmitScanDataToKinesisAsync");
await _fireHoseClient.PutRecordAsync(putRecordRequest);
Console.WriteLine("End EmitScanDataToKinesisAsync");

Upvotes: 0

Related Questions