CODEWITHSUNDEEP
google-bigquerygoogle-cloud-data-fusion
Oussama NEZAR
Oussama NEZAR

Reputation: 9

Update BigQuery table using google datafusion

I've created à pipeline in data fusion with postgres Query as source and BigQuery as Sink. What I wanna do is insert/update on the target table except in one column that I don't wanna update ( insert only ) even if there is a new data on the primary key table. To be precise, the column that I don't wanna update is inserted_at (datetime) so I don't want to lose this information. there is not mush of option in BigQuery Sink to try, and i don't want to do lookup with my main table ( big data in it ) and then split in output.

any leads please?

Thank you

Upvotes: 0

Views: 282

Answers (1)

Pamela Hdz
Pamela Hdz

Reputation: 29

What if you create a function in bigquery to perfom the actions and then in data fusion you only executes the function with the data you want to update in the params? Bigquery is for analytics not a relational database so probably is going to be expensive to perfom. You can add clusterin and partitions to make it cheaper.

Also you can use wrangler to drop the column UPDATED_AT and just update the rest of the columns of your tables..

There are some ideas. Hope it helps

Upvotes: 1

Related Questions