CODEWITHSUNDEEP
azureazure-data-factory
amikm
amikm

Reputation: 65

Creating a dependency pipeline to check file is latest in ADF

I am trying to create a dependency pipeline for files before executing my model refresh (web activity) I want to make sure all the related files are there in their respective folders and all files are latest.

Suppose, my model refreshes uses the following file present in adls-

  1. myadls/raw/master/file1.csv
  2. myadls/raw/dim/file2.csv
  3. myadls/raw/dim/file3.csv
  4. myadls/raw/reporting/file4.csv

We need to compare the files last modified with today's date. If both are equal then files are the latest. If any of the files is not the latest then I need an email with the file name that is not the latest and I shouldn't trigger my web activity which usually does model refresh.

I have created this pipeline using get metadata, for each activity, If-condition, web activity, and Set variable activity. But the problem is I am not able to get an email for the file which is not the latest.

How I can get an email for a file which is not the latest file according to my scenario?

Note, the above folders can have more than 100 files, but I am only looking for specific files I am using in my model.

Upvotes: 1

Views: 309

Answers (2)

Nandan
Nandan

Reputation: 4945

As per your current arch ,you can create variables per foreach activity that would store the file name . So within foreach activity, in case if the file is not latest using append variable activity you can save all file names. and then in the final validation, you can concat all for each loop variables to have the final list of files that are not modified.

But ideally I would suggest the below approach :

  1. Have the list of files created as a lookup activity output.
  2. Provide that to a single foreach activity in sequential execution.
  3. within foreach via IF activity and getmeta data activity, check whether the file is latest or not. If not via append variable activity append the file name.
  4. Once out of foreach, via If condition check whether the file name variable is blank or has some values. If it has values, then you can send an email and the filename variable has all the non updated file names

Upvotes: 0

Trent Tamura
Trent Tamura

Reputation: 1145

We use SendGrid API to send emails at my company.

You can easily pass the FileNames in the body of the email using any email API out there. You can write the FileNames to a variable then reference the variable in the body. It sounds like you have built almost everything out, so within your ForEach, just have an Append to Variable step that writes a new value to your array variable. Then you can use those array values in your SendEmail Web Activity, or use a string conversion function, there are many ways to do it.

I will update this post with example later

Upvotes: 0

Related Questions