CODEWITHSUNDEEP
cloudgoogle-cloud-data-fusion
Amit Kaushik
Amit Kaushik

Reputation: 31

Appending incremental data in Bigquery from MySql using Cloud Data Fusion

I want to schedule a pipeline that transfers MySql data to Google Bigquery, but the complete data gets appended to the old table in BQ, I want only the incremental data to be appended...

Upvotes: 3

Views: 861

Answers (2)

Nitin Motgi
Nitin Motgi

Reputation: 229

You need an ability to 'update if present' schematic which is not currently available in BQ plugin. We are working on updating the plugin to be able to support that (opened a JIRA https://issues.cask.co/browse/CDAP-15298). Alternatively, you can write it to a separate table every run and then use 'merge' to update the primary table. Opened a JIRA to add a merge Action plugin that allows one to merge table (https://issues.cask.co/browse/CDAP-15297)

Upvotes: 2

Ali Anwar
Ali Anwar

Reputation: 431

Is it possible to calculate the diff to import every time based on a column in your data?

Upvotes: 0

Related Questions