How to copy particular files from sFTP source location if the files are not already present in sFTP sink location in Azure Data Factory

Question

I want to filter source folder for files have name starting with 'File'. Then I want to check if those files are already present in sink folder. If not present then copy else skip.

Picture 1 -This is the initial picture which contains files in source and sink

Picture 2 - This is the desired output where only those files are copied which were not present in Sink (except junk files)

Picture 3 - This is how I tried. There are IF & copyData activity in ForEach, But I am getting error in copyData activity.

Answer 1

I have reproed in my local environment as shown below.

1. Get sink files list where filename starts with ‘file’ using Get Metadata activity.

The output of Get Metadata1:

2. Create an array variable to store the list of sink files.

3. Convert Get Metadata activity (Get sink files) output to the array by using ForEach activity and append each filename to an array variable.

@activity('Get Sink Files').output.childItems

Add append variable activity inside ForEach activity.

4. Now get the list of source files using another Get Metadata activity in the pipeline.

The output of Get metadata2:

5. Connect Get Metadata activity2 (Get Source files) output and ForEach activity to another ForEach activity2.

@activity('Get Source Files').output.childItems

6. Add If Condition activity inside ForEach2 activity. Add expression to check the current item (each source file) contains in the array variable.

@contains(variables('sink_files_list'),item().name)

7. When false add copy activity to copy source file to sink.

Source:

Sink: