CODEWITHSUNDEEP
restpaginationazure-data-factoryrelative-url
Vincent Chalmel
Vincent Chalmel

Reputation: 652

Azure Data Factory REST how to manage pagination as a GET url parameter retrieving a pagination id from previous call result

I have an existing data source in Azure Data Factory calling a REST API

My last working config (built with the UI) without pagination uses dataset parameters to build a relative URL, it translates in the source as :

"typeProperties": {
  "relativeUrl": {
    "value": "@concat('theRoute?cursor=',dataset().cursor,'&nb=100&q=aDateParameter:[',dataset().startDate, ' TO ', dataset().endDate, ']')",
    "type": "Expression"
  },
  "requestMethod": "GET",
}

(With the dataset parameter "cursor" default value being "*" to initialize a pagination)

Then in each response I will have to retrieve #.header.nextCursor to get the nextCursor id (# being the json root of the response), and before anybody asks, yes, there is indeed a nested doc named "header" in the json body, nextCursor is not in the http headers.

So I will have to replace in my relative URL the value of the "cursor" parameter with the id in #.header.cursor. But I fails to create the pagination rules. (Even if the doc explicitly list a "Next request’s query parameter = property value in current response body" scenario).

Also I'm wondering if it would simply inject this to replace the actual cursor parameter in my previous relativeURL... Or if I would somehow have to find another way than my concat expression to pass my url parameters for the original relative url (I can't find any documentation on a way to eventually use this QueryParameter syntax outside pagination rules, nor any doc or example for a POST data body dynamic construction syntax)

Upvotes: 0

Views: 3492

Answers (1)

David
David

Reputation: 61

If my understanding is correct, the relative URL for each page looks like 'theRoute?cursor={cursor}&nb=100&q=aDateParameter:[{startDate} TO {endDate}]'.

The {cursor} is from the JSON response body, $.header.nextCursor.

Sorry that the pagination rule today doesn’t support expressions like this, and we are aware of this kind of customer ask. We will look into it to see how to support it in the future.

Upvotes: 1

Related Questions