Reputation: 11
Dynamic BigQuery datasource in Google DataPrep - can it be done?
Can it really be right that I can't define a dynamic datasource using Dataprep. As far as i can see i can only point on a BigQuery Table. I need to point at a certain partion within a table. The naming of the partions always includes the date. like. "ga_20171113".
I really dont understand why i just cant define a SQL thoughwards BQ and use that as a source.
Upvotes: 1
Views: 270
Answers (1)
Reputation: 21
Dataprep can ingest data from BigQuery views, which should hopefully be able to support your use case. https://cloud.google.com/bigquery/docs/views has details on creating views. One note: Dataprep only supports views created using standard SQL rather than legacy SQL. So when creating a view from a query, please make sure that "Use Legacy SQL" is unchecked in the query options.
Upvotes: 2
Related Questions
- Can Google Dataflow connect to API data source and insert data into Big Query
- Change a dataset dynamically on BigQuery
- Dynamically set bigquery dataset in dataflow pipeline
- Google Cloud Dataflow - is it possible to define a pipeline that reads data from BigQuery and writes to an on-premise database?
- Cloud Dataflow: change bigquery destination on the fly
- Add dataset parameters into column to use them in BigQuery later with DataPrep
- Is there a way to create a Bigquery table with data-dependent schema in Google Dataflow?
- Dataflow to set BigQuery partition based on the data
- Google Bigquery query execution using google cloud dataflow
- Dynamic partitioning in google cloud dataflow?