CODEWITHSUNDEEP
azure-data-factory
user10360768
user10360768

Reputation: 225

How to use variable in ADF pipeline's copy activity source

I am creating ADF(V2) pipeline where I have to find the latest folder, based on datetime and copy all the files to destination with the same folder path

for example, if I have source path a/b/c which has folders

20221021010000
20221021020000
20221021030000

When the pipeline runs, I have to copy the latest folder (20221021030000) to destination a/b/c/20221021030000

I already figured out how to find latest folder as per this article, and the variable is being set correctly.

But I don't know how to use this variable in ADF copy's source. I already tried it this way, where I am using list of files and passing the variable

enter image description here

When I run the pipeline, it succeeds. The copy stage gets this as input

{
    "source": {
        "type": "BinarySource",
        "storeSettings": {
            "type": "AzureBlobFSReadSettings",
            "fileListPath": "a/b/c/20221021030000/",
            "deleteFilesAfterCompletion": false
        },
        "formatSettings": {
            "type": "BinaryReadSettings"
        }
    },
    "sink": {
        "type": "BinarySink",
        "storeSettings": {
            "type": "AzureBlobFSWriteSettings"
        }
    },
    "enableStaging": false
}

But nothing gets copied. What would be wrong here?

enter image description here

Upvotes: 1

Views: 2742

Answers (1)

Aswin
Aswin

Reputation: 7156

I tried to repro this. When I set file path type as List of files, and path to the list is given as below, files are not copied from source folder to sink. enter image description here enter image description here

Solution:

  • In order to solve this, I set file path type as file path in dataset.

enter image description here

  • In Source dataset, file path is given as in below image.

src/@{variables('var1')} enter image description here

  • All files from Source are copied to sink folder. enter image description here

Upvotes: 2

Related Questions