CODEWITHSUNDEEP
azure-data-factoryazure-data-lakeu-sql
Tayyab Anwar
Tayyab Anwar

Reputation: 319

Passing properties to U-SQL activity through Azure Data Factory Pipeline through parameters?

I want to pass the slice start/end time properties of a pipeline to a U-SQL activity through the "parameters" property of the activity/pipeline.

The purpose is to generate dynamic file names based on the date the slice starts and orchestrate the process to run daily and process a file of that date.

Is that doable?

Upvotes: 0

Views: 962

Answers (1)

JustLogic
JustLogic

Reputation: 1738

That should be possible. Here is an example of how I am using the slice date to dynamically define folder structure.

"typeProperties": {
                "scriptPath": "script.usql",
                "scriptLinkedService": "LinkedService_AS_Storage",
                "degreeOfParallelism": 3,
                "priority": 100,
                "parameters": {
                    "in": "$$Text.Format('/RawData/{0:yyyy}/{0:MM}/{0:dd}/In.csv',SliceStart)",
                    "out": "$$Text.Format('/TempData/{0:yyyy}/{0:MM}/{0:dd}/Out.csv',SliceStart)"
                }
            }

Inside the USQL script you would just reference the @in or @out parameters. Be careful, U-SQL has case-sensitive syntax. They are actually added to the top of the script file when azure data factory gets it from storage to run.

Hope this helps.

Upvotes: 2

Related Questions