CODEWITHSUNDEEP
google-cloud-platformgoogle-bigquery
WJA
WJA

Reputation: 7004

Trigger cloud function when new data in BigQuery

I would like to trigger a Cloud Function when new data has been imported into a BigQuery table. Ideally, I would like to extract all the rows (one column is ISIN) that have been inserted.

Would this be possible and how?

Upvotes: 16

Views: 16928

Answers (1)

gidutz
gidutz

Reputation: 594

Update June 2022: Cloud Functions 2nd Generation supports Eventarc

Using Eventarc it's possible to listen to a long list of cloud events and then trigger a set of consumers including Cloud Functions.

The combination of serviceName = bigquery.googleapis.com and methodName = jobservice.jobcompleted with additional custom filters can be used to create the Eventrac sink, from which Cloud Functions can be triggered. You can read more here.

This is still not a native BigQuery <> Cloud Function integration, because it uses a 3rd cloud service. Therefore it's not much different than the solution suggested before June 2022 of creating a sink with Cloud Logging. Yet, this is the better practice these days of implementing cross-service triggers on GCP.

Old answer: Unfortunately BigQuery has no triggers as a native feature as of May 2022 :/ However there is a nice workaround you can implement! :)

As you probably know, almost every operation in GCP is logged to Cloud Logging (previously StackDriver Logging). In this article they demonstrate how to trigger a Cloud Function whenever a BigQuery load job finishes. The idea is simple:

  1. Create a filter for BigQuery load jobs in Cloud Logging to catch when load jobs terminate.
  2. Create a sink from Cloud Logging to Pub/Sub topic.
  3. Set your Cloud Function as a consumer on this topic and implement your logic

Google even published this concept as an article.

Upvotes: 17

Related Questions