CODEWITHSUNDEEP
azureazure-data-factory
user1005448
user1005448

Reputation: 637

Azure Data Factory: event not starting pipeline

I've set up a Azure Data Factory pipeline containing a copy activity. For testing purposes both source and sink are Azure Blob Storages.

I wan't to execute the pipeline as soon as a new file is created on the source Azure Blob Storage.

I've created a trigger of type BlovEventsTrigger. Blob path begins with has been set to //

I use Cloud Storage Explorer to upload files but it doesn't trigger my pipeline. To get an idea of what is wrong, how can I check if the event is fired? Any idea what could be wrong?

Thanks

Upvotes: 3

Views: 7353

Answers (6)

Mattheos Protopapas
Mattheos Protopapas

Reputation: 1

An event grid resource provider needs to have been registered, within the specific azure subscription.

Also if you use Synapse Studio pipelines instead of Data Factory (like me) make sure the Data Factory resource provider is also registered.

Finally, the user should have both 'owner' and 'storage blob data contributor' on the storage account.

Upvotes: 0

gunn
gunn

Reputation: 320

And now, on 20.06.2021, same for me: event trigger is not working, though when editing it's definition in DF, it shows all my files in folder, that matches. But when i add new file to that folder, nothing happens!

Upvotes: 1

jschmitter
jschmitter

Reputation: 1949

If you're creating your trigger via arm template, make sure you're aware of this bug. The "runtimeState" (aka "Activated") property of the trigger can only be set as "Stopped" via arm template. The trigger will need to be activated via powershell or the ADF portal.

Upvotes: 0

Berthier Lemieux
Berthier Lemieux

Reputation: 4125

There seems to be a bug with Blob storage trigger, if you have more than one trigger is allocated to the same blob container, none of the triggers will fire.

For some reasons (another bug, but this time in Data factories?), if you edit several times your trigger in the data factory windows, the data factory seems to loose track of the triggers it creates, and your single trigger may end up creating multiple duplicate triggers on the blob storage. This condition activates the first bug discussed above: the blob storage trigger doesn't trigger anymore.

To fix this, delete the duplicate triggers. For that, navigate to your blob storage resource in the Azure portal. Go to the Events blade. From there you'll see all the triggers that the data factories added to your blob storage. Delete the duplicates. enter image description here

Upvotes: 2

Cesar Hernandez
Cesar Hernandez

Reputation: 21

Reiterating what others have stated:

  • Must be using a V2 Storage Account
  • Trigger name must only contain letters, numbers and the '-' character (this restriction will soon be removed)
  • Must have registered subscription with Event Grid resource provider (this will be done for you via the UX soon)
  • Trigger makes the following properties available @triggerBody().folderPath and @triggerBody().fileName. To use these in your pipeline your must map them to pipeline paramaters and use them as such: @pipeline().parameters.paramaetername.

Finally, based on your configuration setting blob path begins with to // will not match any blob event. The UX will actually show you an error message saying that that value is not valid. Please refer to the Event Based Trigger documentation for examples of valid configuration.

Upvotes: 2

Fang Liu
Fang Liu

Reputation: 2361

Please reference this. First, it needs to be a v2 storage. Second, you need register it with event grid. https://social.msdn.microsoft.com/Forums/azure/en-US/db332ac9-2753-4a14-be5f-d23d60ff2164/azure-data-factorys-event-trigger-for-pipeline-not-working-for-blob-creation-deletion-most-of-the?forum=AzureDataFactory

Upvotes: 1

Related Questions