CODEWITHSUNDEEP
azure-data-factory
Sonam
Sonam

Reputation: 83

Execute Data Factory pipeline based on event in table

I have a requirement to trigger Azure Data Factory pipeline whenever there is a new record in a table. Is there any way to achieve this.

Upvotes: 3

Views: 2849

Answers (5)

creedcode
creedcode

Reputation: 13

There's no direct way to do this in ADF (yet). As others have pointed out, you can vote for the feature to be added and Microsoft might add it. But there's still a way to do this:

You can set up a Logic App. There's a really nice video tutorial on this here: https://www.youtube.com/watch?v=z0sMIN4xMSY

You can set up a logic app to use a SQL database as the trigger, and then you can decide if you want to have it trigger based on something new created in a specific table or something modified in a specific table. You can then action the logic app to trigger the run of an Azure data factory pipeline. You can set the Logic App to check the table as often as you like, for example, every minute, 3 minutes or longer etc.

If you want an ADF pipeline to run every time a certain table is modified or row inserted, but the table in question is very large, you can reduce compute by creating a track changes table. Basically have another, very small table that logs changes and then use this table for the one the logic app monitors. This would make it faster and more efficient in such a case.

Give the video a watch.

Upvotes: 0

user12823830
user12823830

Reputation:

We can use logic app to trigger a pipeline based on a query that finds data inserted in past x minutes/seconds

Upvotes: 0

Joel Cochran
Joel Cochran

Reputation: 7728

While this is not currently supported, here's an idea on how to fake it. (Just sharing an idea, not sure whether this is feasible.)

  1. Create a Trigger on INSERT
  2. Trigger executes a Stored Procedure
  3. Stored Procedure uses Polybase to create text file in Blob Storage with the relevant information (like new row ID).
  4. Create a BlobCreated event trigger over that Storage location in ADF or Logic App.

Doing this should end up with an Event Trigger that fires whenever a new row is inserted.

Upvotes: 0

Nandan
Nandan

Reputation: 4925

Currently it is not possible. There is a similar discussion going on in the below thread : https://learn.microsoft.com/en-us/answers/questions/197465/how-to-trigger-an-adf-based-on-any-data-changes-wi.html

Upvotes: 0

Steve Johnson
Steve Johnson

Reputation: 8660

No, Event-based trigger only support Azure Blob Storage by now. You can vote here to progress this feature in Azure Data Factory.

Upvotes: 1

Related Questions