CODEWITHSUNDEEP
azure-data-factory
Umar.H
Umar.H

Reputation: 23099

Reading file names from an azure file_storage directory

I have a file_storage within my azure portal which is roughly like : 

- 01_file.txt
- 02_file.txt
- 03_file.txt

In azure data studio I have a data set which is linked to this file storage.

If possible, I would like to loop through this directory and get a list of all the file names in my ETL Pipeline.

I've had a look at the For Each and look up but I can't figure out how to apply it to the directory.

the end result would be a list of file_names that I would then carry out some further procedures before ingesting the data into azure.

my current work around is to create a JSON file which lists the file_names when I load the data into the file-storage and parse that using look up and For Each but I'd like to know if there is a better solution using datafactory?

Upvotes: 1

Views: 300

Answers (1)

Jay Gong
Jay Gong

Reputation: 23792

Please use GetMetadata-Activity. You could get folder metadata then get file name lists by accessing childItem properties. More details,please refer to https://learn.microsoft.com/en-us/azure/data-factory/control-flow-get-metadata-activity#get-a-folders-metadata

Pipeline configuration:

enter image description here

Execution:

enter image description here

Upvotes: 2

Related Questions