CODEWITHSUNDEEP
google-cloud-platformgoogle-bigquerygoogle-cloud-dataprep
Theorp
Theorp

Reputation: 151

Dataprep - Append data to BigQuery table

I'm using GCP's dataprep to join several csv files with the same column structure, treat some data and then write to a BigQuery database.

I have to record this data in BigQuery. Can I include this data from the dataprep and append them in a BigQuery table?

Upvotes: 0

Views: 1057

Answers (3)

Marcio Lazai Junior
Marcio Lazai Junior

Reputation: 23

My difficultly is in "Connect your data" step. I have plenty of tables with the prefix events_ and I want them all. My intuition is to parameterize doing that events_*, but I do not have this option in bigquery table ingestion.

Step that I want to parametrize

Upvotes: 1

Messier_31
Messier_31

Reputation: 303

Yes, you can include your data from DataPrep and append them in a BigQuery table.

Before running the job, in the "Run Job on DataFlow" section:

  1. Click on the action, since you are using BigQuery for the output, should look like "Create-BigQuery"
  2. In the next windows choose you output table
  3. In the left panel select "Append to this table every run"
  4. Click on update

Now, when you run the Job, this will append your data.

The following documentation can be useful.

Upvotes: 1

y0j0
y0j0

Reputation: 3592

Yes, there is possibility to truncate data or append data to BigQuery table. In the output step DataPrep step in BigQuery table selection you can set that will be appended to table.

enter image description here

Upvotes: 1

Related Questions