CODEWITHSUNDEEP
azure-data-factory
ivanhoe
ivanhoe

Reputation: 15

API access from Azure Data Factory

  1. I want to create a ADF pipeline which needs to access an API and using some filter parameter it will get data from there and write the output in JSON format in DataLake. How can I do that??

  2. After the JSON available in Lake it needs to be converted to CSV file. How to do?

Upvotes: 1

Views: 787

Answers (2)

Yusheng
Yusheng

Reputation: 221

You can create a pipeline with copy activity from HTTP connector to Datalake connector. Use HTTP as the copy source to access the API (https://learn.microsoft.com/en-us/azure/data-factory/connector-http), specify the format in dataset as JSON. Reference https://learn.microsoft.com/en-us/azure/data-factory/supported-file-formats-and-compression-codecs#json-format on how to define the schema. Use Datalake connector as the copy sink, specify the format as Text format, and do some modification like row delimiter and column delimiter according to your need.

Upvotes: 2

Wang Zhang
Wang Zhang

Reputation: 327

the below work follow may meet your requirement:

  1. Involve a Copy activity in ADFv2, where the source dataset is HTTP data store and the destination is the Azure Data lake store, HTTP source data store allows you to fetch data by calling API and Copy activity will copy data into your destination data lake.
  2. Chain an U-SQL activity after Copy activity, once the Copy activity succeeds, it'll run the U-SQL script to convert json file to CSV file.

Upvotes: 0

Related Questions