CODEWITHSUNDEEP
azureazure-data-factoryazure-databricks
wanderingdev
wanderingdev

Reputation: 3

Passing multiple files in one pipeline adf

I'm new to ADF Pipeline. Hello,

I have around n number of files in a nested folder structure as below.

landing/Datasource File1.xml File2.xml ... File..n.xml.

I have to pass each file as a parameter to a Databricks notebook,one after the other on a pipeline For instance, once the file1 is passed as a parameter to a notebook and when the execution of notebook completes , we need to pass file2 as a parameter to the same notebook from adf dynamically.

Example: For a given day lets say 2023-05-13, So, in the time folder(1) there will be 7 files, I will have to pass each file one after the other as a parameter to databricks notebook. Once the timefolder(1) is completed,there will be timefolder(2) and it has some files and we have to pass these files as a prameter to databricks notebook. In this way, for the given date and time folders, we will have to pass each file as a parameter to the notebook dynamically. Note: The notebook and the logic in the notebook is constant.

I used getmetadata activiy and was able to achieve a part where i get all the files in the folder. but unable to design a pipeline for a one at atime pass

Could someone please assist.

Upvotes: 0

Views: 368

Answers (1)

Nandan
Nandan

Reputation: 4935

plz use the get meta data activity to get the list of al files present within the folder. post that use the for each activity to iterate over the number of files (get meta data activity output) within for each , call the databricks activity

sample: https://medium.com/@j.belikova89/azure-data-factory-iterating-over-files-using-getmetadata-8db92926e0be#:~:text=Loop%20through%20the%20Files%3A%20Using,such%20as%20copy%20data%20activities.

Upvotes: 0

Related Questions