CODEWITHSUNDEEP
azure-data-factory
Subhash Ghai
Subhash Ghai

Reputation: 115

Rest API call from copy activity

Hi i am processing a set of ~50K records from a pipe delimeted flat kn azure data factory and need to invoke a rest API call for each input record. So, I am using a foreach loop to access each record and inside the loop, I am using a copy activity to invoke a rest API call. My question is, can I invoke the rest API call in bulk for all the records at once, as the foreach loop is slowing the pipeline execution. I want to remove the foreach loop and also process the API json response and store it in azure sql database.

Thanks

Upvotes: 2

Views: 1675

Answers (1)

Pratik Somaiya
Pratik Somaiya

Reputation: 733

You will have to check the Pagination properties so that you can decide how much payload you need to return from source API:

https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#pagination-support

Also, if you need to store the API JSON response in Azure SQL, then you can do so with many built in functions like JSON_PATH

More details can be found in this link:

https://learn.microsoft.com/en-us/azure/azure-sql/database/json-features

Upvotes: 1

Related Questions