java.lang.NoClassDefFoundError while deserializing protobuf messages

Question

I am using protobuf generated java classes for messages in my dataflow job. I have been trying to get things to run but there is some problem with the dependencies (jar hell) that is causing the deserialization to fail with the following exception:

java.lang.NoClassDefFoundError: Could not initialize class com.brightcove.rna.model.Metadata$MetadataDefaultEntryHolder
    at com.brightcove.rna.model.Metadata.(Metadata.java:50)
    at com.brightcove.rna.model.Metadata.(Metadata.java:13)
    at com.brightcove.rna.model.Metadata$1.parsePartialFrom(Metadata.java:444)
    at com.brightcove.rna.model.Metadata$1.parsePartialFrom(Metadata.java:438)
    at com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:495)
    at com.brightcove.rna.model.Row.(Row.java:79)
    at com.brightcove.rna.model.Row.(Row.java:13)
    at com.brightcove.rna.model.Row$1.parsePartialFrom(Row.java:987)
    at com.brightcove.rna.model.Row$1.parsePartialFrom(Row.java:981)
    at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:192)
    at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:210)
    at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
    at com.google.cloud.dataflow.sdk.coders.protobuf.ProtoCoder.decode(ProtoCoder.java:198)
    at com.google.cloud.dataflow.sdk.coders.protobuf.ProtoCoder.decode(ProtoCoder.java:114)
    at com.google.cloud.dataflow.sdk.runners.worker.WindmillKeyedWorkItem$3.apply(WindmillKeyedWorkItem.java:144)
    at com.google.cloud.dataflow.sdk.runners.worker.WindmillKeyedWorkItem$3.apply(WindmillKeyedWorkItem.java:133)
    at com.google.cloud.dataflow.worker.repackaged.com.google.common.collect.Iterators$8.transform(Iterators.java:817)
    at com.google.cloud.dataflow.worker.repackaged.com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
    at com.google.cloud.dataflow.sdk.repackaged.com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
    at com.google.cloud.dataflow.sdk.repackaged.com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
    at com.google.cloud.dataflow.sdk.repackaged.com.google.common.collect.Iterators$5.hasNext(Iterators.java:569)
    at com.google.cloud.dataflow.sdk.repackaged.com.google.common.collect.Iterators$7.computeNext(Iterators.java:673)
    at com.google.cloud.dataflow.sdk.repackaged.com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
    at com.google.cloud.dataflow.sdk.repackaged.com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
    at com.google.cloud.dataflow.sdk.util.ReduceFnRunner.processElements(ReduceFnRunner.java:281)
    at com.google.cloud.dataflow.sdk.util.GroupAlsoByWindowViaWindowSetDoFn.processElement(GroupAlsoByWindowViaWindowSetDoFn.java:83)
    at com.google.cloud.dataflow.sdk.util.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:49)
    at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.processElement(DoFnRunnerBase.java:138)
    at com.google.cloud.dataflow.sdk.util.LateDataDroppingDoFnRunner.processElement(LateDataDroppingDoFnRunner.java:67)
    at com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:191)
    at com.google.cloud.dataflow.sdk.runners.worker.ForwardingParDoFn.processElement(ForwardingParDoFn.java:42)
    at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerLoggingParDoFn.processElement(DataflowWorkerLoggingParDoFn.java:47)
    at com.google.cloud.dataflow.sdk.util.common.worker.ParDoOperation.process(ParDoOperation.java:53)
    at com.google.cloud.dataflow.sdk.util.common.worker.OutputReceiver.process(OutputReceiver.java:52)
    at com.google.cloud.dataflow.sdk.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:223)
    at com.google.cloud.dataflow.sdk.util.common.worker.ReadOperation.start(ReadOperation.java:169)
    at com.google.cloud.dataflow.sdk.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:69)
    at com.google.cloud.dataflow.sdk.runners.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:645)
    at com.google.cloud.dataflow.sdk.runners.worker.StreamingDataflowWorker.access$500(StreamingDataflowWorker.java:86)
    at com.google.cloud.dataflow.sdk.runners.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:488)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

I am using the following dependencies in my gradle project:

compile 'com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:1.5.0'
compile 'com.google.protobuf:protobuf-java:3.0.0-beta-2'
compile 'io.grpc:grpc-all:0.13.2'

The protobuf files were compiled with the following configuration (using gradle protobuf plugin)

protobuf {
    protoc {
        artifact = 'com.google.protobuf:protoc:3.0.0-beta-2'
    }
    plugins {
        grpc {
            artifact = 'io.grpc:protoc-gen-grpc-java:0.13.2'
        }
    }
    generateProtoTasks {
        all()*.plugins {
            grpc {}
        }
    }
}

UPDATE

Found one more stack trace that may be helpful:

java.lang.NoSuchMethodError: com.google.protobuf.DescriptorProtos$FieldDescriptorProto.hasJsonName()Z
at com.google.protobuf.Descriptors$FieldDescriptor.<init>(Descriptors.java:1208)
at com.google.protobuf.Descriptors$FieldDescriptor.<init>(Descriptors.java:867)
at com.google.protobuf.Descriptors$Descriptor.<init>(Descriptors.java:804)
at com.google.protobuf.Descriptors$Descriptor.<init>(Descriptors.java:568)
at com.google.protobuf.Descriptors$FileDescriptor.<init>(Descriptors.java:474)
at com.google.protobuf.Descriptors$FileDescriptor.buildFrom(Descriptors.java:290)
at com.google.protobuf.Descriptors$FileDescriptor.internalBuildGeneratedFileFrom(Descriptors.java:336)
at com.brightcove.rna.model.Messages.<clinit>(Messages.java:133)
at com.brightcove.rna.model.DimensionSet$Builder.getDescriptorForType(DimensionSet.java:897)
at com.brightcove.rna.model.TransformEvents.populateMessage(TransformEvents.java:28)
at com.brightcove.rna.transforms.StreamingAdditions$1.processElement(StreamingAdditions.java:46)
at com.google.cloud.dataflow.sdk.util.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:49)
at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.processElement(DoFnRunnerBase.java:138)
at com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:191)
at com.google.cloud.dataflow.sdk.runners.worker.ForwardingParDoFn.processElement(ForwardingParDoFn.java:42)
at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerLoggingParDoFn.processElement(DataflowWorkerLoggingParDoFn.java:47)
at com.google.cloud.dataflow.sdk.util.common.worker.ParDoOperation.process(ParDoOperation.java:53)
at com.google.cloud.dataflow.sdk.util.common.worker.OutputReceiver.process(OutputReceiver.java:52)
at com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:161)
at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnContext.outputWindowedValue(DoFnRunnerBase.java:288)
at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnContext.outputWindowedValue(DoFnRunnerBase.java:284)
at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnProcessContext.outputWithTimestamp(DoFnRunnerBase.java:456)
at com.brightcove.rna.transforms.functions.ExtractRawLog.processElement(ExtractRawLog.java:35)
at com.google.cloud.dataflow.sdk.util.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:49)
at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.processElement(DoFnRunnerBase.java:138)
at com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:191)
at com.google.cloud.dataflow.sdk.runners.worker.ForwardingParDoFn.processElement(ForwardingParDoFn.java:42)
at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerLoggingParDoFn.processElement(DataflowWorkerLoggingParDoFn.java:47)
at com.google.cloud.dataflow.sdk.util.common.worker.ParDoOperation.process(ParDoOperation.java:53)
at com.google.cloud.dataflow.sdk.util.common.worker.OutputReceiver.process(OutputReceiver.java:52)
at com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:161)
at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnContext.outputWindowedValue(DoFnRunnerBase.java:288)
at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnProcessContext.output(DoFnRunnerBase.java:450)
at com.google.cloud.dataflow.sdk.transforms.FlatMapElements$3.processElement(FlatMapElements.java:140)
at com.google.cloud.dataflow.sdk.util.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:49)
at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.processElement(DoFnRunnerBase.java:138)
at com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:191)
at com.google.cloud.dataflow.sdk.runners.worker.ForwardingParDoFn.processElement(ForwardingParDoFn.java:42)
at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerLoggingParDoFn.processElement(DataflowWorkerLoggingParDoFn.java:47)
at com.google.cloud.dataflow.sdk.util.common.worker.ParDoOperation.process(ParDoOperation.java:53)
at com.google.cloud.dataflow.sdk.util.common.worker.OutputReceiver.process(OutputReceiver.java:52)
at com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:161)
at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnContext.outputWindowedValue(DoFnRunnerBase.java:288)
at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase$DoFnProcessContext.output(DoFnRunnerBase.java:450)
at com.brightcove.rna.transforms.functions.DecompressDataFn.processElement(DecompressDataFn.java:19)
at com.google.cloud.dataflow.sdk.util.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:49)
at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.processElement(DoFnRunnerBase.java:138)
at com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:191)
at com.google.cloud.dataflow.sdk.runners.worker.ForwardingParDoFn.processElement(ForwardingParDoFn.java:42)
at com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerLoggingParDoFn.processElement(DataflowWorkerLoggingParDoFn.java:47)
at com.google.cloud.dataflow.sdk.util.common.worker.ParDoOperation.process(ParDoOperation.java:53)
at com.google.cloud.dataflow.sdk.util.common.worker.OutputReceiver.process(OutputReceiver.java:52)
at com.google.cloud.dataflow.sdk.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:223)
at com.google.cloud.dataflow.sdk.util.common.worker.ReadOperation.start(ReadOperation.java:169)
at com.google.cloud.dataflow.sdk.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:69)
at com.google.cloud.dataflow.sdk.runners.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:645)
at com.google.cloud.dataflow.sdk.runners.worker.StreamingDataflowWorker.access$500(StreamingDataflowWorker.java:86)
at com.google.cloud.dataflow.sdk.runners.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:488)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

I checked the source for java.lang.NoSuchMethodError: com.google.protobuf.DescriptorProtos$FieldDescriptorProto.hasJsonName()Z and it definitely the hasJsonName() method. For some reason, java isn't able to find it which means it's looking at some other implementation of the class. Is there something that I am missing or something that I can use to figure out why/what is causing the failure?

Answer 1

You seem to be mixing Dataflow SDK for Java, version 1.5.0, with Protocol Buffers runtime version 3.0.0-beta-2 and gRPC version 0.13.2.

Generally speaking, both Protocol Buffers and gRPC don't guarantee compatibility at this time: you'd have to use the runtime that exactly matches the version of the code generation tool that was used at compile-time. Additionally, you can normally have only one version on your Java class path.

Dataflow SDK for Java, version 1.5.0, depends on Protocol Buffers version 3.0.0-beta-1 and gRPC version 0.12.0, and includes code that were compiled with that version of the code generation tool. As such, it would be quite complicated to mix-and-match that with the version that you are currently using.

The best path forward, unfortunately, would be to match the version that the Dataflow SDK is using. Hopefully, we'll be able to do better in Dataflow in the future -- probably around the time these libraries reach full stability and general availability.