CODEWITHSUNDEEP
google-cloud-dataflow
xxtommoxx
xxtommoxx

Reputation: 309

What happens when a branch in the pipeline throws an exception

Let's say for example if my pipeline consumes from Kafka and has two branches. The first branch writes to some data store and the second one produces a count of events seen, both belonging to the same window. What would happen if while making an api request to the datastore it throws an exception, but the second one never does? I.e. Would dataflow stop pulling from Kafka and wait until the first branch recovers or does it keep buffering data since the second one is chugging along fine?

Upvotes: 1

Views: 167

Answers (1)

jkff
jkff

Reputation: 17913

Exceptions are retried.

  • If this is a batch pipeline, it will be retried several times; if it doesn't succeed, the entire pipeline will fail.

  • If this is a streaming pipeline, it will be retried until it succeeds. The rest of the pipeline will continue processing data meanwhile. If the exception keeps happening, you'll need to fix your code and update the pipeline.

Upvotes: 2

Related Questions