CODEWITHSUNDEEP
azure-pipelinesazure-data-factory
kuclimber
kuclimber

Reputation: 43

In ADF V2 - how to append date ("yyyyMMdd")) to filename dynamically for S3 dataset

I'm currently working to automate a pipeline in ADFv2 where the source data sits in S3. A new file is created daily and is structured "data_20180829.csv"

I have tried to instrument dynamic content to accomplish this in the fileName field of Copy Data Activity. However even when I try something as simple as @{concat('data_','20180829.csv')} (that should resolve to the correct value) the source fails.

Is there any way to see what the dynamic content will resolve to?

Upvotes: 4

Views: 11369

Answers (2)

Himanshu Kumar Sinha
Himanshu Kumar Sinha

Reputation: 1776

Just to extend what wBob called out . If you want the structure like foo_YYYY\MM\DD\ , you can always use this 

@concat('foo-name','/',formatDateTime(utcnow(),'yyyy'),'/',formatDateTime(utcnow(),'MM'),'/',formatDateTime(utcnow(),'dd'))

Upvotes: 0

wBob
wBob

Reputation: 14379

This should just be a matter of setting the filename expression in the dataset, eg

Azure Data Factory dataset settings

Note, the setting is done on the dataset not at the Copy activity level. Also note you can make your expression more dynamic by combining the utcnow function with formatDateTime, eg something like this:

@concat('data_',formatDateTime(utcnow(),'yyyMMdd_HHmmss'),'.csv')

Note carefully the case of the formatting strings. Capital MM is for month in a two-digit format. HH is for the hour in 24-hour format. The full list of formatting strings is here.

The json for the dataset looks like this:

{
    "name": "DelimitedText3",
    "properties": {
        "linkedServiceName": {
            "referenceName": "linkedService2",
            "type": "LinkedServiceReference"
        },
        "annotations": [],
        "type": "DelimitedText",
        "typeProperties": {
            "location": {
                "type": "AzureBlobStorageLocation",
                "fileName": {
                    "value": "@concat('data_',formatDateTime(utcnow(),'yyyMMdd_HHmmss'),'.csv')",
                    "type": "Expression"
                },
                "container": "ang"
            },
            "columnDelimiter": ",",
            "escapeChar": "\\",
            "quoteChar": "\""
        },
        "schema": [
            {
                "type": "String"
            },
            {
                "type": "String"
            },
            {
                "type": "String"
            },
            {
                "type": "String"
            }
        ]
    }
}

Upvotes: 5

Related Questions