CODEWITHSUNDEEP
regexazure-data-factory
SubratKP
SubratKP

Reputation: 59

Does Azure Data Factory supports regular expression matching?

In my Azure Data Factory pipeline I have a filter activity where I need to check whether name of a file is matching a specific pattern.

eg: >the file should be a csv file and name should starts with 'D'. >the name of the file must contain a specific word.

These values will come from a config file.

I want the implementation be something like this: My config file will have an attribute called "filePatternt" which will be a regex. In the filter activity, I will extract this attribute and will do something like regex.match(@myregular expression from file pattern attribute, @filename).

But I am not finding any relevant article or regex function that I can use for the above.

Please let me know for any clue or solution or link you have.

Upvotes: 2

Views: 13324

Answers (1)

Leon Yue
Leon Yue

Reputation: 16411

When we copy data from blob to other, Data factory support using some expressions to filter the blobs in wildcard operations, like:

  1. *: If you want to copy all blobs from a container or folder, additionally specify wildcardFileName as *.
  2. *.csv: choose all the csv files from a container or folder.
  3. Start*: copy all blobs from a container or folder which name start with 'Start'.

Ref:

  1. https://learn.microsoft.com/en-us/azure/data-factory/connector-azure-blob-storage#copy-activity-properties
  2. Folder and file filter examples

It may looks like the regex expression, but very simple. enter image description here

Usually, Data Factory doesn't mention about regex expression, we will consider that it's not supported.

For you purpose "the file should be a csv file and name should starts with 'D'. >the name of the file must contain a specific word.", You could using bellow Data Factory Expression in Add dynamic content help you achieve that:

startsWith()
contains()
endsWith()
and()

For example, I just create an example pipeline with Get Metadata and a If Condition. Using bellow condition expression to filter the csv file which name start with 'D' and contains a specific word:

  1. Get metadata to get the file name.
  2. If condition to filter the file.

I prefer to use If condition, the expression is same with Filter conditon:

@and(and(startswith(activity('Get Metadata3').output.itemName,'D'),contains(activity('Get Metadata3').output.itemName,'specifcWords')),endswith(activity('Get Metadata3').output.itemName,'.csv'))

enter image description here

If condition is true(the csv file which name start with 'D' and contains a specific word), we could add the case active.

Upvotes: 3

Related Questions