CODEWITHSUNDEEP
amazon-kinesisamazon-kinesis-firehose
hatellla
hatellla

Reputation: 5142

Multiple Destinations for Kinesis

Can we have multiple destinations from single Kinesis Firehose? I saw this picture enter image description here

From this, it looks like it is possible to add s3, redshift and elastic search from single firehose. I exactly want to do this.

But when I do it from aws console, then it asks for single destination only. For elastic search, it asks for S3 also. So, I am able to add elastic search and s3 but still redhift is left. I am not sure how to do it from same kinesis. Please help.

Upvotes: 12

Views: 12156

Answers (2)

raghavyadav990
raghavyadav990

Reputation: 92

I am currently working on a similar solution. What I found the best way is to first set up the Kinesis stream then add firehose as a consumer/delivery stream. And 1 more consumer as lambda function which can be utilized to send the data to any destination. in my case, it was dynamo DB.

Upvotes: 0

halil
halil

Reputation: 1812

As you said it not available now but, using a recent added feature of firehose, you can write to 3 of them using one firehose.

But i am not sure this is the optimum way of doing this, you need to compare aws cost and development cost then decide.

If you want to try here is how you can do it.

Configure firehose to write to redshift, intermadiate files will be on s3, (possibly intermediate files are deleted) so you can store them in another bucket with bucket replication or triggering new file notification and move it to another bucket using a lambda function. At the same time to write every single record to Elasticsearch, you will use firehose lambda connection, it is in firehose console, called, data transformation. (link below) Write a simple elasticsearch http post write request for every record firehose has got, so you will have data on 3 destination.

https://aws.amazon.com/blogs/compute/amazon-kinesis-firehose-data-transformation-with-aws-lambda/

As i said before this can cost more, and having 2 firehoses doing redshift write and ES+S3 write is more easy.

Upvotes: 6

Related Questions