CODEWITHSUNDEEP
google-cloud-bigtable
Norbert Kremer
Norbert Kremer

Reputation: 21

Google Bigtable export fails with java.lang.NoSuchMethodError

I'm trying to use this process: https://cloud.google.com/bigtable/docs/exporting-sequence-files to export my Bigtable for migration.

I am using Google Cloud Shell, started off of the Google Cloud Platform dashboard. This is Debian GNU/Linux 9.4 (stretch).
java -version reports openjdk version "1.8.0_171"
There does not seem to be a later jre readily available (ie in apt-get) for stretch

I invoke like this:

java -jar bigtable-beam-import-1.3.0-shaded.jar export \
--runner=dataflow \
--project=xx-may23 \
--bigtableInstanceId=xx0523\
--bigtableTableId=test1 \
--destinationPath=gs://xx0523/test1 \
--tempLocation=gs://xx0523/tmp2 \
--maxNumWorkers=3 \
--zone=us-east4-a

Eventually, get this error:

ERROR MonitoringUtil$LoggingHandler:101 - 2018-05-26T02:10:55.387Z: 
java.lang.NoSuchMethodError: 
java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
at com.google.cloud.bigtable.beam.sequencefiles.SequenceFileSink$OutputStreamWrapper.write(SequenceFileSink.java:214)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
at org.apache.hadoop.io.SequenceFile$Writer.writeFileHeader(SequenceFile.java:1145)
at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1214)
at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1091)
at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.<init>(SequenceFile.java:1441)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:274)
at com.google.cloud.bigtable.beam.sequencefiles.SequenceFileSink$SeqFileWriter.prepareWrite(SequenceFileSink.java:157)
at org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:939)
at org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)

It seems to be a java version compatibility issue. How can I fix it?

Upvotes: 2

Views: 191

Answers (1)

xEc
xEc

Reputation: 359

I ran exactly into the same issue while following the documentation. Navigating to the maven repository, I realized there was a newer version, 1.4.0, that you can find here. Invoking the new jar with the command given in the documentation went smoothly and the task succeeded.

Upvotes: 1

Related Questions