Way to trigger dataflow only after Big Query Job finished

Question

actually the following steps to my data:

new objects in GCS bucket trigger a Google Cloud function that create a BigQuery Job to load this data to BigQuery.

I need low cost solution to know when this Big Query Job is finished and trigger a Dataflow Pipeline only after the job is completed.

Obs:

• I know about BigQuery alpha trigger for Google Cloud Function but i
  dont know if is a good idea,from what I saw this trigger uses the job id, which from what I saw can not be fixed and whenever running a job apparently would have to deploy the function again. And of course
  it's an alpha solution.
• I read about a Stackdriver Logging->Pub/Sub -> Google cloud function -> Dataflow solution, but i didn't find any log that
  indicates that the job finished.
• My files are large so isn't a good idea to use a Google Cloud Function to wait until the job finish.

Answer 1

Despite your mention about Stackdriver logging, you can use it with this filter

resource.type="bigquery_resource"
protoPayload.serviceData.jobCompletedEvent.job.jobStatus.state="DONE"
severity="INFO"

You can add dataset filter in addition if needed.

Then create a sink into Function on this advanced filter and run your dataflow job.

If this doesn't match your expectation, can you detail why?

Answer 2

You can look at Cloud Composer which is managed Apache Airflow for orchestrating jobs in a sequential fashion. Composer creates a DAG and executes each node of the DAG and also checks for dependencies to ensure that things either run in parallel or sequentially based on the conditions that you have defined.

You can take a look at the example mentioned here - https://github.com/GoogleCloudPlatform/professional-services/tree/master/examples/cloud-composer-examples/composer_dataflow_examples