CODEWITHSUNDEEP
sslapache-kafkaspark-structured-streaming
Anirban
Anirban

Reputation: 277

Error reading Kafka SSL client truststore file from Spark streaming

I have a Spark streaming application reading from Kafka. I am running it from EMR. Recently I implemented Kafka SSL. I am creating the Kafka client as shown below. I am getting a strange error running the application when it tries to read the truststore file. Error is: 

- Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: java.io.FileNotFoundException: /tmp/kafka.client.truststore.jks (No such file or directory)

What is causing this issue?

DataStreamReader df = session.readStream()
    .format("kafka")
    .option("kafka.bootstrap.servers",kafka_server)
    .option("subscribe", kafka_topic)
    .option("failOnDataLoss", "false")
    .option("group.id", kafka_group)
    .option("kafka.security.protocol","SSL")
    .option("kafka.ssl.truststore.location","/tmp/kafka.client.truststore.jks")
    .option("kafka.ssl.truststore.password","clientpass");

Upvotes: 1

Views: 3064

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191711

That file must exist on the executor nodes of the cluster, not just on the machine you submit the code from

Otherwise, you must pass the --files option to spark-submit to pass them from your machine to the driver, then you could remove /tmp/ part of your string path as well, since the file should be directly accessible

Also note: group.id isn't allowed in structured streaming ; in Spark docs you'll see it's kafka.group.id

Upvotes: 1

Related Questions