Azure Copy Activity to copy Hierarchical JSON from a REST API to Datalake Gen 2 with out explicit mapping

Question

Hi i am trying to copy a JSON object content using a GET request to a file with in data Lake gen 2 and when using Copy Activity i wanted to read all the columns and use the default mapping to load it as a parquet file into data lake gen 2. Could any one show me on what variables to set with in copy activity source as i have multiple REST API calls and I wanted to created separate files for each of them and still use only 1 copy activity to do that? Thank you very much

Some one told me that copy activity is not possible and to use data flow and flatten hierarchy but still in that we need to specify the contents on the JSON to extract all the columns

Answer 1

To copy data from rest API JSON to ADLS gen2 in parquet format follow below procedure:

Create Web Activity to retrieve the details of web activity with API URL and GET method

Create Linked service for Rest API with Base URL:

Create Linked Service for ADLS Gen2 :

connect for each activity on success of web activity enter range function for items expression as @range(1,activity('restAPI').output.total_pages)add copy activity in for each activity, create Rest Api dataset by using created linked service for source of copy activity and set get activity as request method created parameter url and added value as ?page=@{item()} and entered relative url as @dataset().url

Source:

Created ADLS parquet dataset by using created linked service for sink created filename parameter and added Apipage@{item()} as dynamic content and set it to filename in dataset and AFAIK flatten if the JSON is having one array, then it can be flattened in copy activity otherwise need to use data flow.

debug the pipeline, it debugged successfully.

The data is copied successfully to ADLS Gen 2 account.