CODEWITHSUNDEEP
restpaginationazure-data-factory
Anouar
Anouar

Reputation: 85

How to use API offset pagination in Azure COPY ACTIVITY to dynamically stop when the last page has been processed

I'm working on a data integration pipeline in Azure Data Factory, and I need to implement offset pagination for a copy activity. I have an API that returns a response with pagination information, and I want to use that information to perform offset pagination in the copy activity.

Here's an example of the API response:

{
    "log_id": "3133A719-AB38-4846-1111-9D37AD9D682F",
    "pagination": {
        "next_page": 2,
        "last_page": 2428,
        "total_records": 24274,
        "prev_page": 1,
        "current_page": 1
    },
    "data": {
        "ws_fin_out_header": [
            {
                "serial": "TST1400002"
            },
            {
                "serial": "TST1400001"
            }
        ]
    },
    "auth": true,
    "success": true,
    "message": null,
    "uuid": "E0EB1293-1111-4505-A02D-950A6477CF53",
    "version": "4",
    "sqlResult": null,
    "token_succes": true,
    "benchmark": "request finished in 219 ms",
    "token": false
}

I use QueryParameters {pageNumber} = Range(1, 2428,1) but the 2428 is now hardcoded. How do dynamically set the end of the range so that the copy activity stops when all is processed.

I already tried to work with "EndCondition": $.data.ws_fin_out_header = EMPTY. This did not work.

Any ideas how to set the offSet correctly and stop when done?

Upvotes: 0

Views: 1422

Answers (2)

venkat chinta
venkat chinta

Reputation: 1

Based on your Information, I tried it and got error. I have have a total of 38,967 records, and you've set the 'end' expression to $.metadata.total. However, I'm encountering an issue where it's attempting to access the 40th page.enter image description here

Failure happened on 'Source' side. ErrorCode=RestCallFailedWithClientError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Rest call failed with client error, status code 404 NotFound, please check your activity settings. Request URL: https://extapi.winnowsolutions.com/pulldelivery/transactions-datasets/50253/result/?page=40. Response: {"errors":[{"key":"INVALID_DATASET_URL","message":"Invalid dataset url or page number!","source":"command result payloadUrl"}]},Source=Microsoft.DataTransfer.ClientLibrary,'

Upvotes: 0

Pratik Lad
Pratik Lad

Reputation: 8402

To achieve your scenario, you can directly pass the value to end condition and stop pagination in range as below:

In your condition it can be like $pagination.total_records

enter image description here

Also, you can refer this SO thread.

Upvotes: 1

Related Questions