CODEWITHSUNDEEP
apache-beam
Kamal Aman
Kamal Aman

Reputation: 21

How does Apache Beam Fault Tolerance work for Global Windows?

I'm building pipeline with Beam Python. I have a stream of events coming in from PubSub with userId and buttonId. I have a global window that maintains how many times a button was clicked across all users.

If, after some time the server restarts running the Direct Runner/Flink Runner, will the global windows state be restored to the pipeline?

How does the fault tolerance in Beam work?

How does the offset/checkpoint to PubSub get tracked?

The Beam documentation states:

Storage and fault tolerance of state: since state is per-key-and-window, the more keys and windows you expect to process simultaneously, the more storage you will incur".

However, i cannot find any more information on this.

Upvotes: 1

Views: 740

Answers (1)

Ruoyun Huang
Ruoyun Huang

Reputation: 173

To the first part of your question, beam handles exceptions in a streaming service by draining, some details are covered here https://cloud.google.com/dataflow/docs/guides/stopping-a-pipeline

Though not sure if that answers your question on offset/checkpoint.

Upvotes: 0

Related Questions