CODEWITHSUNDEEP
azurecloudazure-data-factorysubdirectoryazure-storage-account
FrMS
FrMS

Reputation: 1

Azure. StorageAccounts, Azure data factory (ADF). Organization of the folders and subfolders

I was wondering if the way me and my group are storing the files is a best practice, if it is not or if it does not matter.

I am reading through PowerBI (PBI) and through Azure data factory (ADF) one folder in a container of a storage account. In that folder there are many files, but also some subfolders with also many files. When I want to read only the files in this folder, but not the subfolder, ADF reads everything by default regardless of whether it is in the folder or in subfolders. In PBI I was able to not read the files in the subfolders but in ADF I have not been able yet...

Should we change this way of storing data or this is not something that really matters? What can be consdiered as best practices? How can I with ADF read only the folder files but not the subfolders?

Any help is welcome, Thx

Upvotes: 0

Views: 89

Answers (1)

Rakesh Govindula
Rakesh Govindula

Reputation: 11529

To avoid copying files from subfolders, you need to uncheck the Recursively option in the copy activity source.

Sample demo:

source container:

enter image description here

Give the path till the required folder. Here, I have given till the source container.

enter image description here

Give the target location as well.

enter image description here

In the source uncheck the Recursively option.

enter image description here

Execute the pipeline and only files from the required main folder will be copied.

enter image description here

Here, I have used binary dataset but the process is same for any kind of dataset.

Upvotes: 0

Related Questions