CODEWITHSUNDEEP
azure-data-factory
Magnus Johannesson
Magnus Johannesson

Reputation: 432

Retrieve blob file name in Copy Data activity

I download json files from a web API and store them in blob storage using a Copy Data activity and binary copy. Next I would like to use another Copy Data activity to extract a value from each json file in the blob container and store the value together with its ID in a database. The ID is part of the filename, but is there some way to extract the filename?

Upvotes: 1

Views: 2275

Answers (2)

tinyMinerva
tinyMinerva

Reputation: 39

After Setting up Dataset for source file/file path with wildcard and destination/sink as some table

  1. Add Copy Activity setup source, sink
  2. Add Additional Columns
  3. Provide a name to the additional column and value "$$FILEPATH"
  4. Import Mapping and voila - your additional column should be in the list of source columns marked "Additional"

Setup

Upvotes: 0

Martin Esteban Zurita
Martin Esteban Zurita

Reputation: 3209

You can do the following set of activities:

1) A GetMetadata activity, configure a dataset pointing to the blob folder, and add the Child Items in the Field List.

2) A forEach activity that takes every item from the GetMetadata activity and iterates over them. To do this you configure the Items to be @activity('NameOfGetMetadataActivity').output.childItems

3) Inside the foreach, you can extract the filename of each file using the following function: item().name

After this continue as you see fit, either adding functions to get the ID or copy the entire name.

Hope this helped!

Upvotes: 1

Related Questions