CODEWITHSUNDEEP
azureazure-data-factory
John11
John11

Reputation: 61

Dynamically load file from blob storage in ADF

I have a multiple parquet files in blob storage. I have to take the latest file from blob and process in ADF (copy activity or data flow) as source. Is there anyway to achieve this.

Upvotes: 0

Views: 1185

Answers (2)

NiharikaMoola
NiharikaMoola

Reputation: 5074

Below are the repro details to get the latest modified file.

  1. Create 2 variables, one to store the latest file name and the second variable to store the last modified date and assign an initial date value (least date) to it.

enter image description here

  1. Using Get Metadata1, get the list of file names.

  2. Pass the Output child items of Get Metadata1 to ForEach activity.

  3. Inside ForEach activity, add another Get Metadata2 to get the last modified date of the current item (@item().name)

enter image description here

  1. Connect Get Metadata2 to the If Condition activity. Add a condition to check if the Get Metdata2 output last modified date is greater than the initial date variable value.

    @greater(ticks(activity('Get Metadata2').output.lastModified),ticks(formatDateTime(variables('initial_date'))))

enter image description here

  1. If the condition satisfies, then add set variable activity inside true to capture the current file name.

enter image description here

  1. Ignore if the condition is false.

  2. The file name which is stored after the ForEach loop is completed is the last modified file. Use this variable (file_name) in your later activities.

Upvotes: 0

Nandan
Nandan

Reputation: 4945

You can use get meta data activity to list down all the files present in source and filter the latest file to be used in copy activity

Upvotes: 0

Related Questions