CODEWITHSUNDEEP
apache-kafkaspark-streaming
Karim Tawfik
Karim Tawfik

Reputation: 1516

Spark streaming 1.6 with kafka 0.11

I have kafka broker upgraded from 0.8 to 0.11, and now I am trying to upgrade the spark streaming job code to be compatible with the new kafka -I am using spark 1.6.2-.

I searched a lot for steps to follow to do this upgrade I didn't find any article either official or not-official.

The only article I found useful is this one, however it is mentioning spark 2.2 and kafka 0.10, but I got a line saying

However, because the newer integration uses the new Kafka consumer API instead of the simple API, there are notable differences in usage. This version of the integration is marked as experimental, so the API is potentially subject to change

Do anyone have tried to integrate spark streaming 1.6 with kafka 0.11, or is it better to upgrade spark first to 2.X , since there is a lack of info about and support regarding this version mix of spark-streaming and kafka?

Upvotes: 2

Views: 526

Answers (1)

Karim Tawfik
Karim Tawfik

Reputation: 1516

After lots of investigations, found no way to do this move, as spark-streaming only supporting kafka version up to 0.10 (which has major differences from kafka 0.11, 1.0.X).

That's why I decided to move from spark-streaming to use the new kafka-streaming api, and simply it was awesome, simple to use, very flexible, and the big advantage is that: IT IS A LIBRARY, you can simply add it to your project, not a framework that is wrapping your code.

Kafka-streaming api almost support all functionality provided by spark (aggregation, windowing, filtering, MR).

Upvotes: 0

Related Questions