Triggering an alert when multiple dataflow jobs run in parallel in GCP

Question

I am using google cloud dataflow to execute some resource intensive dataflow jobs. And at a given time, my system must execute no more than 2 jobs in parallel.

Since each job is quite resource intensive, I am looking for a way to trigger an alert when more than 2 dataflow jobs are running.

I tried implementing a custom_count which increments after the start of each job. But custom_couter only display after the job has executed. And it might be too late to trigger an alert by then.

Answer 1

Another option is to use an GCP monitoring system that is observing the running Dataflow jobs. You can e.g. use Elastic Cloud (available via Marketplace) to load all relevant Metrics and Logs. Elastic can visualize and alert on every state you are interested in. I found this terraform project very helpful in order to get started with that approach.

Answer 2

You could modify the quota dataflow.googleapis.com/job_count of the project to be limited to 1, and no two jobs could run parallel in that project. The quota is at the project level, it would not affect other projects.