CODEWITHSUNDEEP
scalaapache-sparkapache-kafkazstd
Khilesh Chauhan
Khilesh Chauhan

Reputation: 881

Reading Kafka data with zstd compression using scala spark

We are trying to read data from Kafka Producer, with zstd compression.

But getting error -

java.lang.NoClassDefFoundError: com/github/luben/zstd/ZstdOutputStreamNoFinalizer

We are using Spark 3.3.2 and scala 12.2.14. Tried adding -

  1. Jar file - zstd-jni-1.5.2-3.jar
  2. Dependency in pom.xml -
<dependency>
     <artifactId>zstd-jni</artifactId>
     <groupId>com.github.luben</groupId>
     <version>1.5.2-1</version> 
</dependency>

but still same error. Any idea how to resolve this issue?

Upvotes: 0

Views: 102

Answers (1)

jitter
jitter

Reputation: 54615

In the text it says 1.5.2-3 in pom.xml it says 1.5.2-1

Did you try something along the lines of

spark-submit [....] --jars path/to/zstd-jni-1.5.2-3.jar

Also the zstd-jni version is somewhere around ~2 years old and the Spark version is around 18 months old. Did you check if they are compatible?

Upvotes: 0

Related Questions