CODEWITHSUNDEEP
google-cloud-platformgoogle-bigquerygoogle-compute-enginegoogle-cloud-data-fusion
Shubham kumar jain
Shubham kumar jain

Reputation: 31

Data Fusion replication pipeline is not syncing data in Google Bigquery

Hi we want to replicate the data from Mysql(source) to GoogleBigquery(destination) we adopted the method described by google Docs with Data fusion replication pipeline as mentioned in Link https://cloud.google.com/data-fusion/docs/tutorials/replicating-data/mysql-to-bigquery

Berief of what we are doing: Enabling bin log in MY SQL for CDC(Change data Capture) creating a replication pipeline in data fusion starting the pipeline and syncing the data

  1. we are successfully able to create MySql data in comupute engine and enabling bin-log for CDC and provided all necessary permission to user for the data replication pipeline in my SQL
  2. we are successful in creating a data Fusion instance and able to create a replication pipeline
  3. replication pipeline is able to fetch our SQL database details and target Big query is also set
  4. On starting the pipeline it is tracking the Changes successfully (Insert,update and delete ) and table Schema is also created in Bigquery Successfully automatically.
  5. But we are getting PROBLEM that no data is getting transsferred to Bigquery table. In log what i have seen is loading batch of 1 event in to statging Bucket

sharing the screenshot also enter image description here

able to fetch every change from MYSQL but data is not transferring to bigquery table schema was created but data is not transferred loading batch of 1 event in to statging Bucket we are using developer mode and waited for more than 90 mins

Upvotes: 0

Views: 1154

Answers (1)

Sandeep Mohanty
Sandeep Mohanty

Reputation: 1552

The issue might be happening because there may be a schema/data type mismatch with the BigQuery table and the source MYSQL database table on the columns.

For example: if you have a column in source table, in BigQuery this column is of INT64 datatype with a length of 19, while in the source database table, it is Integer type with a length of 10, so you need to update the length of columns as per your datasize.

Upvotes: 1

Related Questions