CODEWITHSUNDEEP
azure-sql-databaseazure-data-factory
GettingItDone
GettingItDone

Reputation: 603

Azure Data Factory: Execute Pipeline activity cannot reference calling pipeline, cyclical behaviour required

I have a number of pipelines that need to cycle depending on availability of data. If the data is not there wait and try again. The pipe behaviours are largely controlled by a database which captures logs which are used to make decisions about processing.

I read the Microsoft documentation about the Execute Pipeline activity which states that

The Execute Pipeline activity allows a Data Factory or Synapse pipeline to invoke another pipeline.

It does not explicitly state that it is impossible though. I tried to reference Pipe_A from Pipe_A but the pipe is not visible in the drop down. I need a work-around for this restriction.

enter image description here

Constraints:

Ideas:

  1. Create a intermediary pipe that can be referenced. This is no good I would need to do this for every pipe that requires this behaviour because dynamic content is not allowed for pipe selection. This approach would also pollute the Data Factory workspace.
  2. Direct control flow backwards after waiting inside the same pipeline if condition is met. This won't work either, the If activity does not allow expression of flow within the same context as the If activity itself.
  3. I thought about externalising this behaviour to a Python application which could be attached to an Azure Function if needed. The application would handle the scheduling and waiting. The application could call any pipe it needed and could itself be invoked by the pipe in question. This seems drastic!
  4. Finally, I discovered an activity Until which has do while behaviour. I could wrap these pipes in Until, the pipe executes and finishes and sets database state to 'finished' or cannot finish and sets the state to incomplete and waits. The expression then either kicks off another execution or it does not. Additional conditional logic can be included as required in the procedure that will be used to set a value to variable used by the expression in the Until. I would need a variable per pipe.

enter image description here

I think idea 4 makes sense, I thought I would post this anyway in case people can spot limitations in this approach and/or recommend an approach.

Upvotes: 1

Views: 3776

Answers (1)

IpsitaDash-MT
IpsitaDash-MT

Reputation: 1430

Yes, absolutely agree with All About BI, its seems in your scenario the best suited ADF Activity is Until :

The Until activity in ADF functions as a wrapper and parent component for iterations, with inner child activities comprising the block of items to iterate over. The result (s) from those inner child activities must then be used in the parent Until expression to determine if another iteration is necessary. Alternatively, if the pipeline can be maintained

The assessment condition for the Until activity might comprise outputs from other activities, pipeline parameters, or variables.

When used in conjunction with the Wait activity, the Until activity allows you to create loop conditions to periodically check the status of specific operations. Here are some examples:

  • Check to see if the database table has been updated with new rows.
  • Check to see if the SQL job is complete.
  • Check to see whether any new files have been added to a specific folder.

Upvotes: 1

Related Questions