CODEWITHSUNDEEP
google-cloud-dataflowapache-beam
Ravi
Ravi

Reputation: 51

Issue while running dataflow

I am getting below error while running dataflow job. I am trying to update my existing beam version to 2.11.0 but I am getting below error at run time.

java.lang.IncompatibleClassChangeError: Class org.apache.beam.model.pipeline.v1.RunnerApi$StandardPTransforms$Primitives does not implement the requested interface com.google.protobuf.ProtocolMessageEnum at org.apache.beam.runners.core.construction.BeamUrns.getUrn(BeamUrns.java:27) at org.apache.beam.runners.core.construction.PTransformTranslation.(PTransformTranslation.java:58) at org.apache.beam.runners.core.construction.UnconsumedReads$1.visitValue(UnconsumedReads.java:49) at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:666) at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649) at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649) at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649) at org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311) at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245) at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458) at org.apache.beam.runners.core.construction.UnconsumedReads.ensureAllReadsConsumed(UnconsumedReads.java:40) at org.apache.beam.runners.dataflow.DataflowRunner.replaceTransforms(DataflowRunner.java:868) at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:660) at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:173) at org.apache.beam.sdk.Pipeline.run(Pipeline.java:313) at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)

Upvotes: 5

Views: 3567

Answers (3)

sha
sha

Reputation: 71

This may be caused by incompatible dependencies. I successfully upgraded beam from 2.2.0 to 2.20.0 by upgrading the dependencies at the same time.

beam.version: 2.20.0
guava.version: 29.0-jre
bigquery.version: v2-rev20191211-1.30.9
google-api-client.version: 1.30.9
google-http-client.version: 1.34.0
pubsub.version: v1-rev20200312-1.30.9

Upvotes: 2

Joseph Zack
Joseph Zack

Reputation: 21

I don't have enough rep to leave a comment, but I ran into this issue and later figured out that my problem was that I had different beam versions in my pom.xml. Some had 2.19 and some had 2.20.

I would do a quick search of your versions in the pom or gradle file to make sure they are all the same.

Upvotes: 2

Kenn Knowles
Kenn Knowles

Reputation: 6023

This usually means that the version of com.google.protobuf:protobuf-java that Beam was built with does not match the version at runtime. Does your pipeline code also depend on protocol buffers?

UPDATE: I have filed https://issues.apache.org/jira/browse/BEAM-6839 to track this. It is not expected.

Upvotes: 1

Related Questions