CODEWITHSUNDEEP
azureazure-data-factoryazure-blob-storage
Florin D. Preda
Florin D. Preda

Reputation: 1378

Azure Data Factory specify custom output filename when copying to Blob Storage

I'm currently using ADF to copy files from an SFTP server to Blob Storage on a scheduled basis.

The filename structure is AAAAAA_BBBBBB_CCCCCC.txt.

Is it possible to rename the file before copying to Blob Storage so that I end up with a folder-like structure like below?

AAAAAA/BBBBBB/CCCCCC.txt

Upvotes: 13

Views: 12983

Answers (3)

William
William

Reputation: 211

Here is what worked for me

I created 3 parameters in my Blob storage dataset, see the image bellow: screenshot of my parameters

I specified the name of my file, added the file extension, you can add anything in the Timestamp just so you could bypass the ADF requirement since a parameter can't be empty.

Next, click on the Connection tab and add the following code in the FileName box: @concat(dataset().FileName,dataset().Timestamp,dataset().FileExtension). This code basically concatenate all parameters do you could have something like "FileName_Timestamp_FileExtension. See the image bellow:

Blob dataset connection tab

Next, click on your pipeline then select your copy data activity. Click on the Sink tab. Find the parameter Timestamp under Dataset properties and add this code: @pipeline().TriggerTime. See the image bellow:

copy data activity

Finally, publish your pipeline and run/debug it. If it worked for me then I am sure it will work for you as well :)

Upvotes: 16

Hauke Mallow
Hauke Mallow

Reputation: 3182

First you have to get the filenames in a GetMetadata-Activity. You can use this as a parameter in a copy-Activity and rename the filenames.

As mentioned in previous answer you can use a replace function to do this:

{
    "name": "TgtBooksBlob",
    "properties": {
        "linkedServiceName": {
            "referenceName": "Destination-BlobStorage-data",
            "type": "LinkedServiceReference"
        },
        "folder": {
            "name": "Target"
        },
        "type": "AzureBlob",
        "typeProperties": {
            "fileName": {
                "value": "@replace(item().name, '_', '\\')",
                "type": "Expression"
            },
            "folderPath": "data"
       }
    },
    "type": "Microsoft.DataFactory/factories/datasets"
}

Upvotes: 0

Fang Liu
Fang Liu

Reputation: 2363

With ADF V2, you could do that. First, use a lookup activity to get all the filenames of your source. Then chain a foreach activity to iterate the source file names. The foreach activity contains a copy activity. Both your source dataset and sink dataset of the cop activity have parameters for filename and folder path. You could use split and replace functions to generate the sink folder path and filename based on your source file names.

Upvotes: 1

Related Questions