CODEWITHSUNDEEP
google-cloud-data-fusioncdap
Korean_Of_the_Mountain
Korean_Of_the_Mountain

Reputation: 1577

Converting data types of Foreign Keys to use Joiner in Google Cloud Data Fusion Pipeline

I am building a pipeline that connects to an on-prem Oracle DB using the Database Plugin, queries two tables (table_a, table_b), and joins those tables using Joiner Plugin, before uploading to a BigQuery table.

The problem I have now is that the Foreign Keys to join table_a and table_b have different data types when I use Get Schema in the Database Plugin. In Joiner, I am joining the tables on table_a.customer_id = table_b.customer_id.

The dtype of table_a.customer_id is LONG but table_b.customer_id is DOUBLE. In the source Oracle DB, both columns are actually integers. For some reason, though, using Get Schema thinks they are LONG and DOUBLE.

I am obviously getting an error in Joiner trying to join on a foreign keys with different data types.

Is there a way to cast/convert the columns from the tables to match so that I can use Joiner?

I've seen some examples using Wrangler Transform to parse dates, but I don't see anything to convert to any other data types. I couldn't find any directive examples either: https://github.com/data-integrations/wrangler.

pipeline

Upvotes: 0

Views: 397

Answers (1)

Edwin Elia
Edwin Elia

Reputation: 399

You can transform your data before joining them by using any of the transform plugins that Cloud Data Fusion offers. As @muscat mentioned, you can use Wrangler transform and utilize the Set type directives, or you can use the Projection transform and configure the Convert field.

Upvotes: 1

Related Questions