CODEWITHSUNDEEP
restazure-data-factory
ALdo
ALdo

Reputation: 111

Rest API Pagination rules in Data Factory

I cannot figure out pagination rule for REST Api in Data factory. Query pulls by default 100 records. I can modify query with parameters ?skip=0&take=5000 and get more records. I cannot see next url in the output. And don't want to hardcode take parameter to 1000000. How can I paginate this call? Thanks. data source

copy task

result

Upvotes: 0

Views: 902

Answers (1)

Pratik Lad
Pratik Lad

Reputation: 8382

According to this MS Document you can use multiple pagination rules to paginate rest API if you don't have the next page URL but you can query the records.

You can follow below pagination rules:

  1. You can use variables in Query parameter/ Absolute URL. Input Skip={offset} either in Base URL or Relative URL as shown in the following screenshots start is similar to skip and limit is similar to take:

enter image description here

And set Pagination rules as either option 1 or option 2：

  • "QueryParameters.{offset}" : "RANGE:0:10000:1000"
  • "AbsoluteUrl.{offset}" : "RANGE:0:10000:1000"

if your Api response contain value which refer to the total no of records you can directly mention it as $node.node in Endcondition.

  1. If you don't want to hard code you can specify the End condition based on
  • value of the specific node in response is empty
  • value of the specific node in response does not exist
  • value of the specific node in response exists
  • value of the specific node in response is a user-defined const value etc.

Upvotes: 1

Related Questions