CODEWITHSUNDEEP
google-bigquerygoogle-cloud-dataprep
Casey Grimes
Casey Grimes

Reputation: 127

Using Dataset with Parameters for BigQuery in Cloud Data Prep?

I have several BigQuery datasets with daily-created tables, such as

etc.

I'd like to set up a schedule Cloud Data Prep job to process these tables each night, so using the dataset with parameters option is really appealing. When I click to replace a BigQuery dataset with dataset with parameters, I get hte following prompt with a BigQuery protocol:

BigQuery path example

Is there a way to use this setup to dynamically pull the right table each day, or do I need to pursue another path such as daily export of the tables to GCS and then using datasets with parameters there?

Upvotes: 0

Views: 292

Answers (1)

Gurkomal
Gurkomal

Reputation: 166

To schedule Cloud Dataprep jobs that process BigQuery tables by date you have the following options:

  1. Use Google Cloud Composer to create a DAG that schedules BigQuery exports to Cloud Storage and then use those exported files with Dataprep Parameters as you have mentioned.
  2. [Recommended] Skip using Dataprep Parameters altogether, and simply create a customized BigQuery view as a Dataprep dataset.

You can then schedule your Dataprep job in either option to run daily.

Concerning option #1, you can set the date range of the Data Prep dataset with parameters so they only accept files within a specified date range.

Upvotes: 2

Related Questions