CODEWITHSUNDEEP
mavenapache-sparkapache-kafkaclassnotfound
Pafvell
Pafvell

Reputation: 1

spark 2.0.2 ClassNotFoundException: org.apache.kafka.clients.consumer.Consumer

Below is my pom.xml. I build the jar with maven shade. I am very sure with the org.apache.kafka.clients.consumer.Consumer is included in my uber jar. Also I have put the kafka-clients-0.10.1.0.jar into spark spark.driver.extraLibraryPath. I also tried add --jars option in spark-submit command. But I still get the classNotFoundException.

            <dependencies>
                <dependency>
                    <groupId>org.scala-lang</groupId>
                    <artifactId>scala-reflect</artifactId>
                    <version>2.11.8</version>
                </dependency>
                <dependency>
                    <groupId>org.apache.spark</groupId>
                    <artifactId>spark-core_2.11</artifactId>
                    <version>2.0.2</version>
                </dependency>
                <dependency>
                    <groupId>org.apache.spark</groupId>
                    <artifactId>spark-streaming_2.11</artifactId>
                    <version>2.0.2</version>
                </dependency>
                <dependency>
                    <groupId>org.apache.spark</groupId>
                    <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
                    <version>2.0.2</version>
                </dependency>
                <dependency>
                    <groupId>org.apache.kafka</groupId>
                    <artifactId>kafka_2.11</artifactId>
                    <version>0.10.1.0</version>
                </dependency>
                <dependency>
                    <groupId>junit</groupId>
                    <artifactId>junit</artifactId>
                    <version>3.8.1</version>
                    <scope>test</scope>
                </dependency>
            </dependencies>

enter image description here

Upvotes: -2

Views: 1144

Answers (2)

Madhumita Bharde
Madhumita Bharde

Reputation: 1

Basically, you need:

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>0.10.1.0</version>
</dependency>

Upvotes: 0

Pafvell
Pafvell

Reputation: 1

I just find a bypass solution. Add the jar into SPARK_HOME/jars. I use spark-submit command. Tried add --jars,--driver-library-path. I am sure the options take effect. But still classNotFound. I find the bypass solution according to the driver log listed as below.

enter image description here

Upvotes: 0

Related Questions