CODEWITHSUNDEEP
google-bigquerycontinuous-integrationcontinuous-deployment
Nilesh Nikumbh
Nilesh Nikumbh

Reputation: 302

CICD pipeline for Bigquery Schema/ ddl deployment

I am looking for CI/CD solution for the Google Bigquery script. The requirement is that I have a list of files with DDL script, design the CI/CD solution which should maintain the version, and deploy the script in Google Bigquery in auto/schedule based.

Upvotes: 0

Views: 1411

Answers (2)

Shipra Sarkar
Shipra Sarkar

Reputation: 1475

Since you want to use version control to commit the schema, you can use CI for Data in BigQuery CLI utility Github Repository which will help you in orchestrating the processes. For more information you can check this documentation. For implementing this, you can check this link.

Since you want to use CD, Cloud Build can be used with BigQuery where you can use your own custom builders for your requirement. You can also configure notifications for both BigQuery and GitHub using Cloud Build.

Upvotes: 2

Bihag Kashikar
Bihag Kashikar

Reputation: 1326

for product recommendation for CI, user cloud source repositories and for CD use cloud build

there are multiple ways to do deploy

option 1: here you are specifying inline query in the cloud build steps, this does not exactly takes into your latest version of SQL. see option 2 for latest version of sql

here you see $PROJECT_ID and $_DATASET these are dynamic variables that you set at run time by environment variables in cloud build also you can use same way to

— name: ‘gcr.io/cloud-builders/gcloud’ entrypoint: ‘bq’ id: ‘create entry min day view’ args: — query — --use_legacy_sql=false — "CREATE OR REPLACE TABLE $PROJECT_ID.$_DATASET.TABLENAME AS SELECT 1"

option 2:

there is post for this here

on the last post above link, you can use use bash as entry point and pass bq arguments as args

hope this helps.

Upvotes: 1

Related Questions