CODEWITHSUNDEEP
apache-kafkaspark-streamingflumehortonworks-data-platform
nilesh1212
nilesh1212

Reputation: 1655

Flafka (Http -> Flume->Kafka ->Spark Streaming)

I have one use case for real time streaming, we will be using Kafka(0.9) for message buffer and spark streaming(1.6) for stream processing (HDP 2.4). We will receive ~80-90K/Sec event on Http. Can you please suggest a recommended architecture for data ingestion into Kafka topics which will be consumed by spark streaming.

We are considering flafka architecture.

Is Flume listening to Http and sending to Kafka (Flafka )for real time streaming a good option?

Please share other possible approaches if any.

Upvotes: 0

Views: 752

Answers (1)

gasparms
gasparms

Reputation: 3354

One approach could be Kafka Connect. Look for a source that fit in your needs or develop a custom new one.

Upvotes: 1

Related Questions