CODEWITHSUNDEEP
javaapache-kafkazstd
MeBigFatGuy
MeBigFatGuy

Reputation: 28598

Apache Kafka 3.8.0 and zstd-jni so load issues

This question is directly related to this previous stackoverflow question"

Upgrading Apache Kafka client to 3.8.0 issue

But i don't see how that answers the question

3.8.0 has added the need to add zstd-jni.jar to the client side. This jar extracts an .so out of that jar to /tmp. However /tmp needs to be executable. If tmp isn't executable,

The advice is to set -DZstdNativePath=some/path to write to somewhere that is.

However, looking at the code, DZstdNativePath expects the path to be the full .so path, as if you manually installed the .so yourself.

So in fact if you just specify a directory, the error message is

java.lang.UnsatisfiedLinkError: Expecting an absolute path of the library: lib

Looking at the code, there is another variable

-DZstdTempFolder=lib

Based on the code, it looks like it will try to extract the .so in this directory.

However, that doesn't seem to happen.

Here is the code responsible for loading that .so

https://github.com/luben/zstd-jni/blob/master/src/main/java/com/github/luben/zstd/util/Native.java

when i set

-DZstdTempFolder=lib

i get

java.lang.UnsatisfiedLinkError: /tmp/libzstd-jni-1.5.6-315290147135917141017.so: /tmp/libzstd-jni-1.5.6-315290147135917141017.so: failed to map segment from shared object no zstd-jni-1.5.6-3 in java.library.path: /usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib Unsupported OS/arch, cannot find /linux/amd64/libzstd-jni-1.5.6-3.so or load zstd-jni-1.5.6-3 from system libraries. Please try building from source the jar or providing libzstd-jni-1.5.6-3 in your system.

Taking a step back, why would 3.8.0 now require this jar, if i am not using compression? If i have to, how do you get this to work?

Upvotes: 0

Views: 1987

Answers (2)

Russell Shaw
Russell Shaw

Reputation: 21

Real solution is to revert back to 3.7.1 until they remove the hard dependency on the lib

Upvotes: 1

MeBigFatGuy
MeBigFatGuy

Reputation: 28598

The answer is the

-DZstdTempFolder=lib

system property was only added in the latest

1.5.6-4

version of zstd-jni.jar.

the 4thparty transient dependency from kafka-clients 3.8.0 was pulling

1.5.6-3

Still don't know why you even need the library if not using compression, but oh well.

Upvotes: 2

Related Questions