CODEWITHSUNDEEP
azureazure-data-factoryazure-synapse
kshitiz sinha
kshitiz sinha

Reputation: 123

Get immediate file name copied using Azure data factory

Scenario: Need to know the filename which was copied from source to destination using copy activity of azure data factory.

Problem : Get metadata can be used for getting the file name. But how do we know which file is copied if more then one file is present in the folder.

Upvotes: 2

Views: 1266

Answers (2)

Nandan
Nandan

Reputation: 4925

You need to follow the below process in case if you want to achieve the requirement via in built ADF activities else it can be easily achieved by python(Azure functions) or csutom activity.

  1. Create 2 variables : MaxLastProcessedDate = 1900-01-01 LatestFile

  2. Use GetMetaDataActivity at folder level to get the list of childItems

  3. Use the output of the childitems field as input to a foreach loop activity

  4. Within Foreach : a) GetMetaDataActivity at file level (since each iteration would be for each file) b) If condition activity (if lastmodifieddate of getmetadata activity > MaxLastProcessedDate)
    for true: update the MaxLastProcessedDate variable to the lastmodifieddate of getmetadata activity and update the LatestFile variable to file name of that iteration.

Upvotes: 0

Steve Johnson
Steve Johnson

Reputation: 8660

Maybe you can try to filter by last modified time. Start time use this expression:

@activity('Copy data1').output.executionDetails[0].start

End time use @utcnow()

enter image description here

Upvotes: 1

Related Questions