CODEWITHSUNDEEP
google-cloud-dataflow
Enzhong Wang
Enzhong Wang

Reputation: 23

Trigger dataflow job when new data arrived in Bigquery

I have two BQ table A and B. What I want to achieve is when new data arrive in table A, a dataflow job can be triggered to transformed the new record and insert into table B, is this possible? If yes, is there any best practice?

Upvotes: 1

Views: 737

Answers (1)

Abdel Megahed
Abdel Megahed

Reputation: 1

I'm sure there are multiple ways of solving this problem. One way derived from this stakoverflow thread is as follows:

  1. Push Stackdriver Logs to Pub/Sub and apply a strong filter that only lets through logs of BigQuery load jobs into table A. Check out the docs here for more information.
  2. Create a Cloud Function instance with a Pub/Sub trigger.
  3. The responsability of that function instance is to kickstart a Dataflow Batch job, which will carry out the necessary transformation before finally writing the resulting PCollection into table B.

Upvotes: 0

Related Questions