CODEWITHSUNDEEP
azure-data-factoryazure-data-explorerfile-copying
Morten Knudsen
Morten Knudsen

Reputation: 1

Copy subdirs + files from storage acct to ADX using Data Factory

I'm trying to copy files from Az Storage blob to ADX using Data factory, but I can't find a solution to do this using json datasets (not binary), so I can map schema. I would love to have a template to do this.

I have tried to follow the resolution mentioned here (Get metadata from Blob storage with "folder like structure" using Azure Data Factory pipeline), but I'm lacking some more guidance (this is my first project using ADF)

enter image description here

enter image description here

enter image description here

Upvotes: 0

Views: 130

Answers (3)

Kranthi Pakala
Kranthi Pakala

Reputation: 1424

From the error message you have shared, it seems like your dynamic expression for passing the childItems of your get metadata activity to ForEach activity items is causing this problem.

You might be using Items = @activity('GetMetadataActivity').output instead of @activity('GetMetadataActivity').output.childItems.

Please use Items = @activity('GetMetadataActivity').output.childItems in your ForEach activity which should help resolve your error.

Here is a video demonstration by a community volunteer where this error has been explained in detail: ADF Error Function Length expects its parameter to be array or string

Upvotes: 0

Morten Knudsen
Morten Knudsen

Reputation: 1

It seems as I was going in a wrong direction.

I have actually found a simple solution for setting this up. I have tried the copy data tool, and it seems to be doing what I want. So case closed :-)

Upvotes: 0

Morten Knudsen
Morten Knudsen

Reputation: 1

Now I am getting another error as shown below

I'm actually also looking for a COMPLETE guide setting this up.

Here is my overall use-case target https://learn.microsoft.com/en-us/azure/sentinel/store-logs-in-azure-data-explorer?tabs=azure-storage-azure-data-factory - but the documentation is missing the detailed steps - in step 6 Create a data pipeline with a copy activity, based on when the blob properties were last modified. This step requires an extra understanding of Azure Data Factory. For more information, see Copy activity in Azure Data Factory and Azure Synapse Analytics.

enter image description here

enter image description here

Upvotes: 0

Related Questions