CODEWITHSUNDEEP
performanceazure-data-factoryuntil-loop
Geezer
Geezer

Reputation: 497

Azure Data Factory Until loop slow to end/terminate

We have an until loop in a ADFv2 pipeline.

The time it takes to stop/terminate once the expression condition is met seems to corrolate between the length of time the until loop takes to completes its activities.

This particular until loop performs alot of activites and can take anywhere between 90-120 mins to complete. So it takes almost as long to end/terminate (break out of the loop).

If I "hack" it so that it only performs a handful of activities it will quickly end and break once it's finished and the expression to terminate is met.

It's like a spinning wheel that keeps spinning even after the power is turned off. The momentum that was built up while connected takes a while to slow down and eventually stop.

Is this a known issue, how can I troubleshoot the exact cause here or fix it?

Upvotes: 1

Views: 419

Answers (1)

mujiannan
mujiannan

Reputation: 41

Incorrect usage of nested loop switch could cause this.

Here is a Until component, and some activities after it:
Until

In the Until, some activities,
Correct:
In until - correct
Incorrect (Slow to end/terminate):
In until - incorrect

Why?

For the incorrect case, the last activity If waiting depends on three activities, perhaps its behavior was counterintuitive.

// pay attention to "dependsOn"
{
    "name": "If waiting",
    "type": "IfCondition",
    "dependsOn": [
        {
            "activity": "Set loop_waiting_refresh_status to True",
            "dependencyConditions": [
                "Succeeded"
            ]
        },
        {
            "activity": "WeChatEP_Notifier Info Get new bearer",
            "dependencyConditions": [
                "Succeeded"
            ]
        },
        {
            "activity": "Set loop_waiting_refresh_status to False",
            "dependencyConditions": [
                "Succeeded"
            ]
        }
    ],
    ...
}

Upvotes: 1

Related Questions