CODEWITHSUNDEEP
google-cloud-dataflow
Venkatesh Chitnis
Venkatesh Chitnis

Reputation: 33

Reseting system lag for cloud dataflow streaming job

Per definition, System Lag is the maximum time that an element is awaiting processing. Assume that dataflow is consuming messages from cloud PubSub. If there happens to be a message which isn't acknowledged for some reason, the System Lag keeps going up (just because of that 1 message). Is there any way to reset the System Lag for a streaming job without terminating the job?

Upvotes: 2

Views: 429

Answers (1)

Andy Xu
Andy Xu

Reputation: 101

You can't reset the System Lag manually. The System Lag is an indication of how healthy your pipeline is. A large System Lag, say caused by the one message you mentioned, can help you see the pipeline issue quickly. Once the blocking message is processed, the System Lag will be updated with a new value showing current pipeline state.

Upvotes: 1

Related Questions