CODEWITHSUNDEEP
azure-data-factory
Graham
Graham

Reputation: 7802

How to increment a parameter in an Azure Data Factory Until Activity?

I am accessing a RESTful API that pages results in groups of 50 using the HTTP connector. The REST connector doesn't seem to support Client Certificates so I can't use the pagination in that.

I have a Pipeline Variable called SkipIndex that defaults to 0. Inside the Until loop I have a Copy Data Activity that works (HTTP source to BLOB sink), then a Set Variable Activity that I am trying to get to increment this Variable.

{
    "name": "Add 50 to SkipIndex",
    "type": "SetVariable",
    "dependsOn": [
        {
            "activity": "Copy next to temp",
            "dependencyConditions": [
                "Succeeded"
            ]
        }
    ],
    "userProperties": [],
    "typeProperties": {
        "variableName": "SkipIndex",
        "value": {
            "value": "50++",
            "type": "Expression"
        }
    }
}

Everything I have tried results in errors such as "The expression contains self referencing variable. A variable cannot reference itself in the expression." and the one above with 50++ causes a sink error during debug.

How can I get the Until loop to increment this variable after it retrieves data?

Upvotes: 9

Views: 14836

Answers (1)

Jay Gong
Jay Gong

Reputation: 23782

Agree that REST Connector does supports pagination but does not for Client Certificates Authentication type.

For the idea of your Until activity scenario,i am tripped by the can't self-reference a variable in an expression limitation also. Maybe you could make a little trick on that: Add one more variable to persist the index number.

For example,i got 2 variables: count and indexValue

Until Activity:

enter image description here

Inside Until Activity:

enter image description here

V1:

enter image description here

V2:

enter image description here

BTW, no usage of 50++ in ADF.

Upvotes: 12

Related Questions