CODEWITHSUNDEEP
azurepaginationazure-data-factorydata-pipeline
Rubin Shah
Rubin Shah

Reputation: 1

Azure Data Factory - Retrieve next pagination link (decoded) from response headers in a copy data activity of Azure Data Factory

I have created a copy data activity in azure data factory and this data pipeline pulls the data from an API (via REST activity source) and writes the response body (json) on a file kept in the azure blob storage. The API which I am fetching the response from, is paginated and the link to next page is sent in the response headers in response->headers->link.

This URL to next page is in the following general format: <https%3A%2F%2FsomeAPI.com%2Fv2%2FgetRequest%3FperPage%3D80%26sortOrder%3DDESCENDING%26nextPageToken%3DVAdjkjklfjjgkl>; rel="next"

I want to fetch the next page token present in the above URL and use it in the pagination rule.

I have tried using some pagination rules:

> AbsoluteURL = Headers.link But, this did not work as the entire encoded link shown above, is getting appended directly and hence, the pipeline throws an error.

I have also tried to use the query parameters but could not get any result.

I have followed questions on Stack Overflow and have read the documentation:

https://learn.microsoft.com/en-us/answers/questions/874836/adf-copy-activity-using-parameter-for-query

https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#pagination-support

How can I access this next page token? What pagination rule is needed support the scenario?

Pasting postman output and ADF data pipeline, for reference.

Postman Response Headers Output:
Postman Response Headers Output

Pagination Rules, I need help on:
Pagination Rules, I need help on

Upvotes: 0

Views: 1412

Answers (2)

Pratik Lad
Pratik Lad

Reputation: 8382

AFAIK, there is no direct option to decode the URL in ADF paginating rule.

You can follow below approach:

  • First take a set variable and create temp variable as string and value as 1 enter image description here

  • Then take until activity and give the condition as @equals(variables('temp1'),10) so it will iterate over it till condition matches. enter image description here

  • Under the until loop take another set variable and create variable URL1 as string and value as URL of rest api. enter image description here

  • After this pass variable value to copy activity and copy the first page to destination. enter image description here

  • After this take web activity and pass that URL variable to it and get response headers. enter image description here

  • After this take set variable and update the URL to next page url usnig dynamic expression.

    @uriComponentToString(split(split(activity('Web1').output.ADFWebActivityResponseHeaders["link"],'>')[0],'<')[1])

    enter image description here

  • In last create a variable called increment as string and increment initial temp variable for every iteration by @add(int(variables('temp1')),1) enter image description here

By this process you can copy as many pages as you want or you set in Until loop as we don't have any specific condition to terminate loop.

Upvotes: 0

Bertek
Bertek

Reputation: 9

My team adopted Pratik's approach with one caveat, we set variable 3 to be outside of the Until activity. Otherwise we would keep overriding url1 value every run.

Our ADF pipeline: Our ADF pipeline

Upvotes: 0

Related Questions