CODEWITHSUNDEEP
azureazure-data-factory
Michelle Santos
Michelle Santos

Reputation: 267

how to segregate files in a blob storage using ADF copy activity

I have a copy data activity in ADF and I want to segregate the files into a different container based on the file type.

ex. Container A - .jpeg, .png Container B - .csv, .xml and .doc

My initial idea was to use 'if condition' and 'or' statement but looks like my approach won't work. I'd appreciate it if you could give some inputs.

enter image description here

Upvotes: 0

Views: 715

Answers (2)

NiharikaMoola
NiharikaMoola

Reputation: 5074

First, get the list files from the source container and loop each file in the Foreach activity to check the extension using If condition and copy files based on the condition to their respective containers.

I have the below files in my source container.

enter image description here

In ADF:

  1. Using the Get metadata activity, get the list of all files from the source container.

enter image description here

  1. Output of Get Metadata activity:

enter image description here

  1. Pass the output list to Foreach activity.

@activity('Get Metadata1').output.childitems

enter image description here

  1. Inside the Foreach activity, add If Condition activity to separate the files based on extension.

@or(contains(item().name, '.xml'),contains(item().name, '.csv'))

enter image description here

  1. If the condition is true, copy the current file to container1.

enter image description here

enter image description here

  1. If the condition returns false, copy the current file to container2 in False activity.

enter image description here

enter image description here

Files in the container after running the pipeline.

Container1:

enter image description here

Container2:

enter image description here

Upvotes: 1

Anand Sowmithiran
Anand Sowmithiran

Reputation: 2920

You should use the GetMetadata activity to first get all the file types that needs to be first passed to a CopyData activity which copies to Container A, and then add next Getmetadata activity to get file types for next copydata activity that copies to container B. so, your ADF pipeline may be like GetMetaData1 - > Copydata1 -> GetMetaData2 -> Copydata2. Refer how to use GetMetaData activity in this article, and documentation

Copy activity itself allows more than one wildcard path, so you could use that in the Data source, see this.

Upvotes: 1

Related Questions