CODEWITHSUNDEEP
azureazure-data-factory
Alejandro Alvarez
Alejandro Alvarez

Reputation: 144

Azure Data Flow source from output in pipeline

Is it possible to create a Data Flow that gets data from the output of an activity in a Pipeline?

Something like this: enter image description here

I want to create a pipeline that has an Azure function activity to process some data and then pass that processed data to a Data flow.

But apparently the Data flow only accepts a Dataset as a source.

Upvotes: 0

Views: 821

Answers (2)

RithwikBojja
RithwikBojja

Reputation: 11363

If your Azure Function Activity gives a array of Json, Then you can follow below process:

Here I have used Lookup Instead of Azure Function which gives Following output:

enter image description here

Use this Output in ForEach Activity and generate array for name object like below with append variable activivty inside ForEach:

 [ "Rakesh", "Bhavan", "sai", "kiran", "kowsik", "Rakesh", "kowsik", "arun", "sai", "virat" ]

Now create a dataflow and with an array parameter. Pass the above array to the Dataflow Parameter from Pipeline:

enter image description here

Inside DataFlow take a source with dummy dataset and dummy column, Generate the column from the array parameter using derived column expression unfold($arr) like below:

enter image description here

Derived Column Result with column:

enter image description here

But the above procedure only works for a single array. So, if you have multiple arrays you have to store the Azure function Output to a blob or ADLS and then Get it back to Dataflow as suggested by @Ziya Mert Karakas.

Upvotes: 2

Ziya Mert Karakas
Ziya Mert Karakas

Reputation: 901

I dont think this is possible as the data needs to be stored somewhere. My advice would be to put the data somewhere from the function and take it from there to the dataflow. ADF doesnt have the capacity on its own to store data.

Upvotes: 0

Related Questions