CODEWITHSUNDEEP
azureazure-data-factory
Simon B
Simon B

Reputation: 1096

How can I variably name an Azure Data Factory file being written to Blob Storage?

I've created an ADF with the wizard which copies data from Azure SQL to Blob storage.

At the option to specify an output filename (see screenshot), I'd like to append to the filename, the date the file was created or some other variable value. Is there a way to do this?

ADF Screenshot

Upvotes: 2

Views: 7268

Answers (2)

Ulf Åkerstedt
Ulf Åkerstedt

Reputation: 3236

Besides using the Data Factory Copy Wizard you can also change the file path and/or file name of a blob file by editing the json configuration of the OutputDataset that controls the file creation, (after having set up the copy pipeline) like this:

  1. Go to the OutputDataset table source (json configuration). To do this you can, for instance,
    1. Open the Data Factory blade for your Data Factory.
    2. Under Contents click Datasets to open the Datasets blade.
    3. Click the OutputDataset whose naming you want to adjust. This opens the OutputDataset blade.
    4. Click the Table Source tile. You are directed to json configuration settings for the OutputDataset.
  2. Add or adjust variables under the partitionedBy property as in the example below. (See e.g. "firstCustomVariable".)
  3. Adjust the fileName and/or the filePath to use the variables according to your demands as in the example below. (See e.g. "blobFileName-{firstCustomVariable}.json.gz".)

See also:
V1: Azure Data Factory - Functions and System Variables
V2: System variables supported by Azure Data Factory
V2: How to read or write partitioned data in Azure Data Factory version 2
Custom Date and Time Format Strings

Example:

{
    "name": "OutputDataset-abc",
    "properties": {
        "published": false,
        "type": "AzureBlob",
        "linkedServiceName": "Destination-BlobStorage-abc",
        "typeProperties": {
            "fileName": "blobFileName-{firstCustomVariable}.json.gz",
            "folderPath": "folderName-{secondCustomVariable}",
            "format": {
                "type": "JsonFormat",
                "filePattern": "arrayOfObjects"
            },
            "partitionedBy": [
                {
                    "name": "firstCustomVariable",
                    "value": {
                        "type": "DateTime",
                        "date": "SliceStart",
                        "format": "yyyyMMddHH"
                    }
                },
                {
                    "name": "secondCustomVariable",
                        "value": {
                        "type": "DateTime",
                        "date": "SliceStart",
                        "format": "MM"
                    }
                }
            ],
            "compression": {
                "type": "GZip",
                "level": "Fastest"
            }
        },
        "availability": {
            "frequency": "Day",
            "interval": 1
        },
        "external": false,
        "policy": {}
    }
}

Upvotes: 2

Joyce Xu
Joyce Xu

Reputation: 26

Yes, it's supported to specify variables on ADF Copy Wizard. The supported variables are: {year}, {month}, {day}, {hour}, {minute} and {custom}. See Data Movement Activities article for details about these variables. Example: inputfolder/{year}/{month}/{day}.

Upvotes: 1

Related Questions