CODEWITHSUNDEEP
mysqlgoogle-cloud-platformgoogle-bigqueryreplicationgoogle-cloud-data-fusion
Mk_cloud
Mk_cloud

Reputation: 21

Replicating data from MySQL to BigQuery using GCP Data Fusion - Getting issue with 'Date' datatype

I wanted to replicate Mysql tables held in GCP Compute Engine to the GC BigQuery. I referred this document : https://cloud.google.com/data-fusion/docs/tutorials/replicating-data/mysql-to-bigquery. so I Decided to use GCP Data Fusion for the Job.

Everything works fine and the Data is replicated in Bigquery. So I was testing different datatype support for this Replication.

Where I come up with issue in this Replication Pipeline, So whenever I try to put the 'DATE' datatype Column for the Data fusion replication, the whole table (Which contain 'DATE' Column) doesn't show up in BigQuery

It Creates the table with schema same as source and 'Date' datatype also present in Bigquery, and I have use the same Date format as supported by BigQuery.

I also gone through Data fusion logs, It shows pipeline is Loading the data perfectly fine into BigQuery, Also catches the new rows added into Mysql Table from source Mysql DB with inserts and updates as well. But somehow rows are not getting into Bigquery.

Did anyone used Data fusion Replication with 'Date' column Datatype ? Is this issue with BigQuery or Data Fusion ? Do I need to provide any manual setting in the BigQuery ? Can anyone please provide inputs on this ?

Upvotes: 1

Views: 842

Answers (2)

Mk_cloud
Mk_cloud

Reputation: 21

I'll mark this issue as Resolve. Since the Problem is with Data Fusion, and the latest version 6.4.1 now supports Datatime datatype while replicating in BigQuery. I'm Receiving correct date & datetime data now.

Thank you for all the Help :)

Upvotes: 1

Sagar Kapare
Sagar Kapare

Reputation: 196

I used following schema which had Date field in it.

create table tutorials_tbl(tutorial_id INT NOT NULL AUTO_INCREMENT, tutorial_title VARCHAR(100) NOT NULL, tutorial_author VARCHAR(40) NOT NULL, submission_date DATE, PRIMARY KEY ( tutorial_id ));

When I run the replication pipeline, I see BQ table is created with following schema:

enter image description here

I also see the events in the table:

enter image description here

Can you please share the input table schema? You can also check the Job History and Query History tabs under BQ table to see if any errors. enter image description here

Upvotes: 0

Related Questions