CODEWITHSUNDEEP
azure-data-factorypipelineazure-synapse
Volume999
Volume999

Reputation: 102

Can you use a data flow sink as a source in the same data flow?

I am trying to load the sales data to the database using Azure Synapse Analytics pipelines, and the strategy is as follows (scenario is made up):

  1. Load the students` data to the table Students
  2. Load the students` classes information to the table StudentsClasses. In this data flow I need to join the data with the Students table (obviously, the new data about students must be loaded to Students at this join step)

Can I have these two processes in the same data flow with Sink ordering? Or does the sink ordering not define source read ordering? (that is, the source reading and transformations are done in parallel, and only the write is according to the ordering?

Edit: This is an example data flow that I want to implement: enter image description here

source3 and sink1 are the same table. What I want is to first populate sink1, then use it for source 2 to join with it. Can this be implemented using Sink ordering? Or source3 will be empty regardless of sink ordering?

Upvotes: 1

Views: 3036

Answers (1)

KarthikBhyresh-MT
KarthikBhyresh-MT

Reputation: 5034

Yes, you can use multiple source and sinks in a single data flow and reference same source over join activity. And order sink write using Custom sink ordering property

I am using Inline dataset but you can use any type

Using inline dataset to store the result in sink1. In source3, use the same inline dataset to join with Source2

enter image description here

enter image description here

Make sure you give the sink order correctly, if you have the wrong order or if it encounters no data while proceeding with transformation, it will publish with no errors however the pipeline run would fail.

enter image description here

Refer MS DOC: Sink ordering

Upvotes: 1

Related Questions