CODEWITHSUNDEEP
azuretriggersazure-data-factory
David Gard
David Gard

Reputation: 12047

Azure Data Factory V2 using yesterdays date when triggering a pipeline run

I have an Azure Data Factory (v2) that I use to backup the contents of a database to a blob store nightly (2am UTC). However, I expect the name of the file to contain the day of the month (dd) that the backup was generated, but it's always the day before.

The file name is generated using an expression -

@{formatDateTime(pipeline().parameters.windowStart,'dd')}.json

So for example the run at 3am today should have been called 23.json, but it was actually called 22.json. 3am is the expected run time as I'm in the UK, which is currently on BST (UTC+1)

Looking at the parameters of the run, I can see that the windowStart is indeed a day out. For example, todays run which was triggered at 2am on the 23rd had 9/22/2020 2:00:00 AM.

Is anybody able to explain why Data Factory is behaving in this way, and hopefully how I can make it work as expected.

Here is the trigger as exported from the Data Factory.

{
    "name": "Trigger_Copy_Transactions",
    "properties": {
        "annotations": [],
        "runtimeState": "Started",
        "pipeline": {
            "pipelineReference": {
                "referenceName": "Copy_Transactions",
                "type": "PipelineReference"
            },
            "parameters": {
                "windowStart": "@trigger().outputs.windowStartTime"
            }
        },
        "type": "TumblingWindowTrigger",
        "typeProperties": {
            "frequency": "Hour",
            "interval": 24,
            "startTime": "2020-08-24T02:00:00Z",
            "delay": "00:00:00",
            "maxConcurrency": 50,
            "retryPolicy": {
                "intervalInSeconds": 30
            },
            "dependsOn": []
        }
    }
}

Upvotes: 0

Views: 1360

Answers (1)

Vitali Dedkov
Vitali Dedkov

Reputation: 318

One thing you could try is to force the file to be generated in the same time zone that your IR is running in. For example we have a Self Hosted IR so when we would generate files it would not match EST times. In that case I did the following:

@concat('File_name',formatDateTime(
convertFromUtc(utcnow(),'Eastern Standard Time'),'yyyy-MM-dd'),'.txt')

Perhaps doing that would force the proper date?

Are you also using Auto-generated IR or Self-Hosted IR when running this job?

Upvotes: 1

Related Questions