CODEWITHSUNDEEP
azure-data-factory
elasticSol
elasticSol

Reputation: 195

Azure Data Factory V2 Copy Activity - Save List of All Copied Files

I have pipelines that copy files from on-premises to different sinks, such as on-premises and SFTP. I would like to save a list of all files that were copied in each run for reporting.

I tried using Get Metadata and For Each, but not sure how to save the output to a flat file or even a database table.

Alternatively, is it possible to fine the list of object that are copied somewhere in the Data Factory logs?

Thank you

Upvotes: 1

Views: 2566

Answers (1)

Joseph Xu
Joseph Xu

Reputation: 6043

Update:

Items:@activity('Get Metadata1').output.childItems enter image description here

If you want record the source file names, yes we can. As you said we need to use Get Metadata and For Each activity.
I've created a test to save the source file names of the Copy activity into a SQL table.

  1. As we all know, we can get the file list via Child items in Get metadata activity. enter image description here The dataset of Get Metadata1 activity specify the container which contains several files. enter image description here The list of file in test container is as follows:
    enter image description here

  2. At inside of the ForEach activity, we can traverse this array. I set a Copy activity named Copy-Files to copy files from source to destnation. enter image description here

  3. @item().name represents every file in the test container. I key in the dynamic content @item().name to specify the file name. Then it will sequentially pass the file names in the test container. This is to execute the copy task in batches, each batch will pass in a file name to be copied. So that we can record each file name into the database table later. enter image description here

  4. Then I set another Copy activity to save the file names into a SQL table. Here I'm using Azure SQL and I've created a simple table.

create table dbo.File_Names(
    Copy_File_Name varchar(max)
);

  1. As this post also said, we can use similar syntax select '@{item().name}' as Copy_File_Name to access some activity datas in ADF. Note: the alias name should be the same as the column name in SQL table. enter image description here

  2. Then we can sink the file names into the SQL table.
    enter image description here Select the table which created previously.
    enter image description here

  3. After I run debug, I can see all the file names are saved into the table. enter image description here

If you want add more infomation, you can reference the post I maintioned previously.

Upvotes: 3

Related Questions