CODEWITHSUNDEEP
azureazure-data-factory
Obiii
Obiii

Reputation: 834

Copy files in subdirs to azure storage with ADF

I have a folder structures like this:

folder1/folder2
               /YearNumber1
                           /monthYear1
                                     /somefile.csv, tbFiles.csv
                           /monthYear2
                                     /somefile2.csv, tbFiles2.csv
                           ...(many folders as above)
               /YearNumber2
                           /montYear11
                                      /somefileXXYYZz.csv, otherFile.csv
                           /monthYear12
                                       /someFileRandom.csv. dedFile.csv
                           ...(many folders as above)

Source: Binary, linked via fileshare linked service

Destination: Binary, on azure blob storage

I don't want to retain the structure, just need to copy all csv files.

Using CopyActivity:

Wildcard Path: @concat('folder1/folder2/','*/','*/',) / '*.csv'

with recursive

But it copies nothing, 0 Bytes.

Upvotes: 0

Views: 987

Answers (1)

KarthikBhyresh-MT
KarthikBhyresh-MT

Reputation: 5044

You can use the below options in the CopyActivity Source Setting:

1. File path type

Allowed wildcards are: * (matches zero or more characters) and ? (matches zero or single character); use ^ to escape if your actual folder name has wildcard or this escape char inside.
See official MS docs for more examples in Folder and file filter examples.

  • wildcardFolderPath - The folder path with wildcard characters under your file system configured in dataset to filter source folders.
  • wildcardFileName - The file name with wildcard characters under your file system + folderPath/wildcardFolderPath to filter source files.

2. recursive - When set to true the data is read recursively from the subfolders.

Example:

enter image description here

If there are only .csv files in your source directories you can simply specify wildcardFileName as just *

Upvotes: 1

Related Questions