CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-data-fusion
Savannah
Savannah

Reputation: 103

Google Cloud Data Fusion - Dynamic arguments based on functions

Good morning all,

I'm looking in Google Data Fusion for a way to make dynamic the name of a source file stored on GCS. The files to be processed are named according to their value date, example: 2020-12-10_data.csv

My need would be to set the filename dynamically so that the pipeline uses the correct file every day (something like this: ${ new Date(). Getfullyear()... }_data.csv

I managed to use the arguments in runtime by specifying the date as a string (2020-12-10) but not with a function.

And more generally is there any documentation on how to enter dynamic parameters with ready-made or custom "functions" (I couldn't find it)

Thanks in advance for your help.

Upvotes: 2

Views: 1217

Answers (1)

narendra solanki
narendra solanki

Reputation: 412

There is a readymade workaround, you can give a try "BigQuery Execute" plugin.

Steps:

  1. Put below query in SQL

select cast(current_date as string) ||'_data.csv' as filename --for output '2020-12-15_data.csv'

  1. Row As Arguments to 'true'
  2. Now use the above arguments via ${filename} wherever you want to.

Upvotes: 1

Related Questions