CODEWITHSUNDEEP
google-cloud-dataflowapache-beam
user3777228
user3777228

Reputation: 159

Google Dataflow Apache beam version upgrade fails if we update and existing pipeline

I have a Google dataflow streaming pipeline running on Apache beam Java 2.50.0 . I wish to upgrade to 2.56.0 (current latest) via the update pipeline option. However , the update gives and error -

The new job is not compatible with 2024-06-17_11_26_33-885445509533346704. The original job (if it existed) has not been aborted. To verify whether the replacement job is compatible with the old job, before you launch the new job, validate its job graph by passing --dataflow_service_options=graph_validate_only. See https://cloud.google.com/dataflow/docs/guides/updating-a-pipeline#validate Causes: The Coder or type for step Read PubSub Events/PubsubUnboundedSource has changed

There has been no code change from my side. If I start a new pipeline with 2.56.0 it starts successfully . However , I need to update an existing running pipeline only.

Upvotes: 0

Views: 314

Answers (1)

陳約翰
陳約翰

Reputation: 11

If you're submitting a replacement job that uses a later version of the Apache Beam SDK, set --updateCompatibilityVersion to the Apache Beam SDK version used in the original job.

Try adding --updateCompatibilityVersion=2.50.0 and see if it works. For me it works from 46 to 56, but it doesnt work from 43 to 56...

Upvotes: 1

Related Questions