CODEWITHSUNDEEP
google-bigquerygoogle-cloud-data-fusion
user16436399
user16436399

Reputation: 1

Google Cloud Data Fusion, How can I load many tables to bigquery in one pipeline

I want to load many tables which is in aws rds mysql server by using cloud data fusion. each table storage is more than about 1gb. also I found the plugin which name is "multiple database table" to load multi table. but i got a fail. Also basically when I used database source I can check my tables' schema. However, in multiple database table, i can 't find how to check table's schema. how can i use this plugin? or is there any other way to load many tables in data fusion service?

My pipeline setting was as follows. enter image description here

Upvotes: 0

Views: 1858

Answers (1)

PjoterS
PjoterS

Reputation: 14084

I'm posting this Community Wiki as OP didn't provide enough details to reproduce but the below information might help someone.

There are few ways to get your data using Cloud Data Fusion, you can use pipeline, plugin, driver and a few others depending on your needs.

On the internet you can find two very well described guides with examples.

If you would like to find some information about Cloud Data Fusion with GCP products you should read Bahadir Bulut guide - How I used Google Cloud Data Fusion to create a data warehouse - Part 1 and Part 2. Also Data Fusion allows to use 150+ preconfigured connectors and transformations like Amazons S3, SQS, etc. Azure services and many more.

Another well described (which I guess would help OP) is to configure both Amazon and GCP resources and using pipelines. This guide is Building a Simple Batch Data Pipeline from AWS RDS to Google BigQuery — Part 1: Setting UP AWS Data pipeline and second part Building a Simple Batch Data Pipeline from AWS RDS to Google BigQuery — Part 2: Setting up BigQuery Transfer Service and Scheduled Query.. In short this guide describes 2 main steps:

  1. Extract data from MYSQL RDS and bring into S3 using AWS data pipeline service
  2. From S3, bring the file inside Bigquery using BigqQuery transfer service.

Upvotes: 1

Related Questions