CODEWITHSUNDEEP
pythonapache-sparkpysparkapache-kafkaspark-streaming
Ahmed
Ahmed

Reputation: 65

Pyspark 3.1.1 direct streaming with kafka?

Pyspark 2.4.7 contains the ability to create a direct stream listener to a kafka topic (documentation)

However, the 3.1.1 (latest) version of pyspark doesn't have this feature. And the latest documentation for kafka direct streaming doesn't include python examples anymore.

My question is how to use direct streaming (not structured streaming) with kafka and pyspark 3.1.1?

Upvotes: 1

Views: 970

Answers (1)

user15869840
user15869840

Reputation: 26

kafka 0.8 support is deprecated as of Spark 2.3.0.

spark-streaming-kafka-0-8 has language support for Scala, Java, Python but spark-streaming-kafka-0-10 does not support Python.

Please refer the below link

https://spark.apache.org/docs/2.4.7/streaming-kafka-integration.html

Upvotes: 1

Related Questions