CODEWITHSUNDEEP
amazon-web-servicesstreamingamazon-kinesisamazon-kinesis-firehose
user51961
user51961

Reputation: 31

Is Kinesis Firehose a replacement to Kinesis Streams?

Kinesis Firehose, as well as Kinesis Streams, are used to load streaming data as per the details mentioned in the AWS blogs. There is no concept of shards or maintenance in case of Firehose. In such a case, Is Kinesis Firehose a replacement to Kinesis Streams?

Upvotes: 3

Views: 2711

Answers (5)

Daniel
Daniel

Reputation: 15393

I am not sure how it would be a replacement if there is no persistence of data with Kinesis Firehose, unless you mean it in the context of there is no need for data persistence, or perhaps it's an issue of cost, then your option would be to analyze that data as soon as it comes in which is Kinesis Firehose and eventually storing it in S3 or Elasticsearch Cluster.

Upvotes: 0

Ashan
Ashan

Reputation: 19728

Kinesis Firehose is not a replacement to Kinesis Streams although there are several use cases, Kinesis Firehose has taken over after its introduction.

Kinesis Streams is used to buffer the streaming data from producers and streaming it into custom applications for data processing and analysis which will consume the temporary buffered stream data.

Data producers push data to Kinesis Streams -> Applications read the data from stream and process.

Kinesis Firehose is used to capture and load streaming data into other Amazon services such as S3 and Redshift so that analysis can take place later on.

Data producers push data to Kinesis Firehose -> Data Transformation using Lambda -> Store in S3 or Redshift.

These two can also be used in combination where, Kinesis Streams can stream the data in to Kinesis Firehose so that, it could be persisted after processing.

Upvotes: 4

Sharhabeel Hamdan
Sharhabeel Hamdan

Reputation: 1549

No, just different purposes.

With Kinesis Streams, you build applications using the Kinesis Producer Library put the data into a stream and then process it with an application that uses the Kinesis Client Library and with Kinesis Connector Library send the processed data to S3, Redshift, DynamoDB or ElasticSearch.

With Kinesis Firehose it’s a bit simpler where you create the delivery stream and send the data to S3, Redshift or ElasticSearch (using the Kinesis Agent or API) directly and storing it in those services.

Kinesis Streams, on the other hand, can store the data for up to 7 days.

You may use Kinesis Streams if you want to do some custom processing with streaming data. With Kinesis Firehose you are simply ingesting it into S3, Redshift, DynamoDB or ElasticSearch.

Upvotes: 0

vicentefuenzalida
vicentefuenzalida

Reputation: 101

A thing to take into account when choosing which service to use are the limits and scalability of each solution. AWS Firehose has a fixed limit of 5mb/sec or 5000 rec/sec (details here), although it can be increased by contacting AWS through a request form. On the other hand, AWS Kinesis can be scaled easily by increasing the number of shards for each Stream (up to 500 shards by default). The main issue here is that each shard has its own cost and you can only scale up or down by doubling the current amount of shards.

As Ashan said, these services serve different purposes, but you can use each one on its own, or combine them according to your needs. The main advantage here, is that Kinesis Stream can be consumed by many consumers, and be fed by many producers. On the other hand, Firehose Streams act as a consumer for other source of data (such as a Kinesis Stream) and can output data to only one destination (S3, Redshit, Elasticsearch, Splunk).

Upvotes: 0

John Rotenstein
John Rotenstein

Reputation: 269091

Amazon Kinesis Firehose is an easy way to create a stream where data is sent to one of:

  • Amazon S3
  • Amazon Redshift
  • Amazon Elasticache

You can also create a Lambda function that can manipulate the data on the way through.

If the above suits your needs, then Firehose could be considered a replacement for Kinesis Streams. However, Kinesis Streams offers more flexibility so it is not an exact replacement.

Upvotes: 5

Related Questions