Azure Data Factory 2 : How to split a file into multiple output files

Question

I'm using Azure Data Factory and am looking for the complement to the "Lookup" activity. Basically I want to be able to write a single line to a file.

Here's the setup:

• Read from a CSV file in blob store using a Lookup activity
• Connect the output of that to a For Each
  • within the For Each, take each record (a line from the file read by the Lookup activity) and write it to a distinct file, named dynamically.

Any clues on how to accomplish that?

Answer 1

Use Data flow, use the derived column activity to create a filename column. Use the filename column in sink. Details on how to implement dynamic filenames in ADF is describe here: https://kromerbigdata.com/2019/04/05/dynamic-file-names-in-adf-with-mapping-data-flows/

Answer 2

Data Flow would probably be better for this, but as a quick hack, you can do the following to read the text file line by line in a pipeline:

1. Define your source dataset to output a line as a single column. Normally I would use "NoDelimiter" for this, but that isn't supported by Lookup. As a workaround, define it with an incorrect Column Delimiter (like | or \t for a CSV file). You should also go to the Schema tab, and CLEAR the schema. This will generate a column in the output named "Prop_0".

2. In the foreach activity, set the Items to the Lookup's "output.value" and check "Sequential".

3. Inside the foreach, you can use item().Prop_0 to grab the text of the line:

4. To the best of my understanding, creating a blob isn't directly supported by pipelines [hence my suggestion above to look into Data Flow]. It is, however, very simple to do in Logic Apps. If I was tackling this problem, I would create a logic app with an HTTP Request Received trigger, then call it from ADF with a Web activity and send the text line and dynamic file name in the payload.