CODEWITHSUNDEEP
apache-sparksslapache-kafkaspark-streaming
Suresh
Suresh

Reputation: 39501

Enabling SSL between Apache spark and Kafka broker

I am trying to enable the SSL between my Apache Spark 1.4.1 and Kafka 0.9.0.0 and I am using spark-streaming-kafka_2.10 Jar to connect to Kafka and I am using KafkaUtils.createDirectStream method to read the data from Kafka topic.

Initially, I got OOM issue and I have resolved it by increasing the Driver memory, after that I am seeing below issue, I have done little bit of reading and found out that spark-streaming-kafka_2.10 uses Kafka 0.8.2.1 API, which doesn't support SSL (Kafka supports SSL only after 0.9.0.0 versions). are there any alternatives to enable SSL between Spark 1.4.1 and Kafka 0.9.0.0.

Here is the log

iableProperties: Property security.protocol is not valid
16/10/24 18:25:09 WARN utils.VerifiableProperties: Property ssl.truststore.location is not valid
16/10/24 18:25:09 WARN utils.VerifiableProperties: Property ssl.truststore.password is not valid
16/10/24 18:25:09 INFO utils.VerifiableProperties: Property zookeeper.connect is overridden to 
16/10/24 18:25:09 INFO consumer.SimpleConsumer: Reconnect due to error

Exception in thread "main" org.apache.spark.SparkException: java.io.EOFException
org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:99)

Upvotes: 1

Views: 2814

Answers (1)

Suresh
Suresh

Reputation: 39501

Support for Kafka 0.10 (which has SSL support) has been added to Spark 2.0. we have to use maven artifact spark-streaming-kafka-0-10_2.10.

Upvotes: 4

Related Questions