CODEWITHSUNDEEP
azureazure-data-factoryazure-databricksazure-data-explorerazure-data-lake-gen2
MMV
MMV

Reputation: 208

ingest multiple json files from ADLS gen2 to ADX through ADF

I'm new to both of ADF(Azure Data Factory) and ADX(Azure Data Explorer). I have multiple Json files in ADLS in different folder level, and I need to ingest all the files into ADX.

ex) UserData/Overground/UsersFolder/project1/main/data/json/demo-02/2021/01/28/03/demo-02-2021-01-28-03-30.json
UserData/Overground/UsersFolder/project1/main/data/json/demo-02/2021/01/28/04/demo-02-2021-01-28-03-30.json
UserData/Overground/UsersFolder/project1/main/data/json/demo-02/2021/01/29/03/demo-02-2021-01-28-03-30.json
UserData/Overground/UsersFolder/project1/main/data/json/demo-02/2021/02/23/03/demo-02-2021-01-28-03-30.json

I'm just wondering if I need to create as many tables in ADX as the number of the Json files in ADLS.. so if I have 1000 Json files in ADLS, should I create 1000 tables in ADX to copy the data from adls to adx?
and how could I copy the data from adls to adx in ADF?

Appreciate your help in advance

Upvotes: 0

Views: 331

Answers (1)

Abhishek Khandave
Abhishek Khandave

Reputation: 3240

To copy from multiple folders, you can use Additional settings of Copy activity Source. For more information follow this official document. You may need to use wildcard for multiple files.

Additional settings:

recursive Indicates whether the data is read recursively from the subfolders or only from the specified folder. Note that when recursive is set to true and the sink is a file-based store, an empty folder or subfolder isn't copied or created at the sink.

Allowed values are true (default) and false. This property doesn't apply when you configure fileListPath.
Also refer Azure Data Explorer as Sink

Azure Data Explorer is supported as a source, where data is copied from Azure Data Explorer to any supported data store, and a sink, where data is copied from any supported data store to Azure Data Explorer. Integrate Azure Data Explorer with Azure Data Factory

Upvotes: 0

Related Questions