CODEWITHSUNDEEP
azureazure-data-factory
enavuio
enavuio

Reputation: 1500

Azure Data Factory Copy Activity New Last Modified Column from Metadata

I am copying many files into one with ADF Copy Activity but I want to add a column and grab the Blob's Last modified date on the Metadata like the $$FILEPATH.
enter image description here

enter image description here

Is there an easy way to do that as I only see System Variables related to pipeline details etc.

https://learn.microsoft.com/en-us/azure/data-factory/control-flow-system-variables

Upvotes: 0

Views: 1995

Answers (1)

Saideep Arikontham
Saideep Arikontham

Reputation: 6114

  • Since the requirement is to add a column to each file where this column value is the lastModified date of that blob, we can iterate through each file, add column to it which has the current blob's lastModified date, copy it into a staging folder.

  • From this staging folder, you can use final copy activity to where you merge all the files in this folder to a single file in the final destination folder.

  • Look at the following demonstration. The following are my files in ADLS storage.

enter image description here

  • I used Get Metadata to get the name of files in this container (final and output1 folders are created in later stages, so they won't affect the process).

enter image description here

  • Using the return filenames as items (@activity('Get Metadata1').output.childItems) in the for each activity, I obtained the lastModified of each file using another get metadata activity inside the for each.

enter image description here

  • The dataset of this Get Metadata2 is configured as shown below:

https://i.sstatic.net/6bErm.png

  • Now I have copied these files into output1 folder by adding an additional column where I gave the following dynamic content (lastModified from get metadata2)
@activity('Get Metadata2').output.lastModified

enter image description here

  • Now you can use a final copy data activity after this foreach to merge these files into a single file into the final folder.

enter image description here

  • The following is the final output for reference:

enter image description here

Upvotes: 1

Related Questions