Get custom metadata for blob files in Azure data factory

Question

I have few 100 files in a folder in Blob Storage. Each of the files have custom metadata (Dictionary type). So when traversing through all files I need to get those metadata of each files.

So how to read that details. I tried using GetMetadata feature which has some hardcoded features like, exists, filename, lastedit etc. But I need to get the custom metadata of those files.

Please share some ideas.

Answer 1

Andreas' answer is great! To expand a little on it: To the Get Metadata activity, we can add a custom property that will appear in the json output, called userProperties. It contains a string of all the custom blob metadata separated by commas. Each value has the metadata name as a string, the equal sign, and a base64 string of the contents, which can include other equal signs.

Using a foreach activity, we can iterate on every custom metadata tag:

@split(activity('metadataActivityName').output.userProperties, ',')

Then add an If condition activity to check that the item matches the desired tag name:

@equals(split(item(), '=')[0], 'customTag')

And finally, in the true case, add a desired activity, like Set Variable and set a pipeline variable to the value from the custom metadata. Here it is important to not use split on the equal sign because equal signs are possible in the base64 encoded string!:

@decodeBase64(replace(item(),'customTag=', ''))

• Replace customTag and metadataActivityName with the corresponding values in your case.

Answer 2

You can actually get the custom metadata of the files in question, but you need to jump through a few hoops. In the example below, I have a custom metadata property called maxmodifieddate set on a specific file. In your Get Metadata activity, go to the Settings tab and select Add dynamic content. Enter the value userProperties:

This will return your userProperties array, but your values will be Base64 encoded:

You can use the decodeBase64 function to retrieve your variable value, like so:

@decodeBase64(replace(activity('Get Metadata1').output.userProperties, 'MaxModifiedDate=', ''))

Which will finally output your metadata value (in this case saved to a variable called myvar):

Answer 3

@Sandeep

I am assuming you're looking to get the user-defined Metadata through Get Meta data Feature.

Unfortunately, this is currently not possible - the activity only returns the predefined set of Metada.

The list of Metadata supported has been documented here.

Workaround

One of the workarounds I can think is make use of the Web Activity in the ADF and hit the REST API Get Blob Properties

The Get blob properties API returns all user-defined metadata, standard HTTP properties, and system properties for the blob.