CODEWITHSUNDEEP
apirestpaginationazure-data-factory
aldo23
aldo23

Reputation: 49

Azure Data Factory Pagination with offset

I have a project scenario to get all users from an endpoint URL and insert it to a SQL table, I am using ADF Pipeline but I'm having some issues with pagination, I have used before pagination but only with nextpage:"someurl" using AbsoluteUrl but now I am having issues because my response comes as follow and for me to get next pages I need to perform multiple calls as follow.

BaseURL: https://xyz.xyz.com/api/v2/users?take=500 which gives me the first 500 Response:

> ],
"total": 3059,
"next": 500
}

In order for me to take the next 500 rows, I need to make requests as follow

And so on... Note skip workes as offset in my case and I have 36000 rows to fetch.

Could you give me any recommendations on how to approach this? Thank You in advance.

Upvotes: 0

Views: 2281

Answers (1)

Giri
Giri

Reputation: 91

Please check if below solution works,

  1. Define a variable in the main pipeline let's say offset with value 0

  2. Inside Until activity
    a. have a web call with dynamic content as below - @concat('https://xyz.xyz.com/api/v2/users?take=500&skip=',variables('offset'))

    b. Perform required operation on data - like writing response to ADLS

    c. use "set variable" activity to increase offset value by 500.

  3. Use proper expression in your Until loop when to exit, there are number of ways to do that, below is one of the approaches - just set a variable when you get zero records in your web call response, in your case - https://xyz.xyz.com/api/v2/users?take=500&skip=36000 use this variable in until expression.

Upvotes: 1

Related Questions