CODEWITHSUNDEEP
azureazure-data-factory
SuperNova
SuperNova

Reputation: 109

how to pass a folder from a data lake store as a parameter to a pipeline?

In data factory, I know you can pass a parameter at the beginning of a pipeline, and then access it later using @pipeline(). If I have a folder in a data lake store, how can I pass that as a parameter and have access to it later (let's say I want to loop a for-each over each file inside it.) Do I pass the path to the folder? Am I passing it as an object?

Upvotes: 1

Views: 1203

Answers (2)

Fang Liu
Fang Liu

Reputation: 2363

  1. First, you need create a data lake store linked service. It will contain the path of azure data lake store. You could use azure data factory UI to create the linked service

  2. Then you need create a data lake store dataset reference that linked service in step 2.

  3. Then you create a getMetaData activity reference dataset in step 2.

  4. Then following steps provided by summit.

All of these can be done in UI.https://learn.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory-portal#create-a-pipeline

Upvotes: 1

Sumit B
Sumit B

Reputation: 300

Here are the steps that you can use -

  1. You can use pass folder path as a parameter (string) to the pipeline.

  2. Use the path and "Get Metadata" activity with "Child Items". This will return the list of files in JSON Format

    Get Metadata Selection

  3. Loop through using "Foreach" activity and perform any action.

    Use output from metadata activity as Items in Foreach activity (example below)

@activity('Get List of Files').output

Hope this helps

Upvotes: 1

Related Questions