CODEWITHSUNDEEP
apache-kafkaapple-m1snappy
jss
jss

Reputation: 229

org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Mac and os.arch=aarch64

I'm building a cdc pipeline to read mysql binlog through maxwell and putting them into kafka my compression type is snappy in maxwell config.But at consumer end in my spring project I'm getting this error.

org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Mac and os.arch=aarch64
at org.xerial.snappy.SnappyLoader.findNativeLibrary(SnappyLoader.java:361) ~[snappy-java-1.1.7.7.jar:1.1.7.7]
at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:195) ~[snappy-java-1.1.7.7.jar:1.1.7.7]
at org.xerial.snappy.SnappyLoader.loadSnappyApi(SnappyLoader.java:167) ~[snappy-java-1.1.7.7.jar:1.1.7.7]
at org.xerial.snappy.Snappy.init(Snappy.java:69) ~[snappy-java-1.1.7.7.jar:1.1.7.7]
at org.xerial.snappy.Snappy.<clinit>(Snappy.java:46) ~[snappy-java-1.1.7.7.jar:1.1.7.7]
at org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:435) ~[snappy-java-1.1.7.7.jar:1.1.7.7]
at org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:466) ~[snappy-java-1.1.7.7.jar:1.1.7.7]
at java.base/java.io.DataInputStream.readByte(DataInputStream.java:271) ~[na:na]
at org.apache.kafka.common.utils.ByteUtils.readUnsignedVarint(ByteUtils.java:170) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.common.utils.ByteUtils.readVarint(ByteUtils.java:205) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.common.record.DefaultRecord.readFrom(DefaultRecord.java:296) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.common.record.DefaultRecordBatch$2.doReadRecord(DefaultRecordBatch.java:278) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.common.record.DefaultRecordBatch$StreamRecordIterator.readNext(DefaultRecordBatch.java:617) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:582) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:551) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.nextFetchedRecord(Fetcher.java:1578) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.fetchRecords(Fetcher.java:1613) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.access$1700(Fetcher.java:1454) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:687) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:638) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1299) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1233) ~[kafka-clients-2.7.2.jar:na]
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1206) ~[kafka-clients-2.7.2.jar:na]

My java version "17.0.2" 2022-01-18 LTS and using Apple M1 chip.

Upvotes: 8

Views: 14487

Answers (2)

Tranfer Will
Tranfer Will

Reputation: 178

I am using kafka client and meet the same error. I fixed it by download snappy-java jar and put it in /libs.

One version of snappy-java maybe here: https://repo1.maven.org/maven2/org/xerial/snappy/snappy-java/1.1.10.7/

Upvotes: 0

Son H. Nguyen
Son H. Nguyen

Reputation: 406

You'll need to update snappy version to 1.1.8.4 or later:

    <dependency>
      <groupId>org.xerial.snappy</groupId>
      <artifactId>snappy-java</artifactId>
      <version>1.1.8.4</version>
    </dependency>

Source: https://issues.apache.org/jira/browse/BEAM-11863?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&focusedCommentId=17318180#comment-17318180

Upvotes: 12

Related Questions