ADF doesnt create BlobCreation event with dataflow

Question

I have pipeline and a dataflow activity inside which copies the data to blob storage. I have trigger activated.

Problem is, the trigger works If I place the file manually on storage. But it doesn't get triggered when the dataflow puts file on the blob storage with copy activity.

Here is the trigger info:

Answer 1

I ended up using web activity to send custom blob events to custom events and using custom triggers on the receiving pipeline.

Answer 2

The problem is that a sink in dataflow when using parquet format generates a BlobRenamed event instead of BlobCreation. Therefore, the trigger doesn't get the right event.

Answer 3

I tried and it's working fine for me. it is detecting blob is getting added in particular container.

It appears that the trigger is set up to only react when new files are added to the blob storage and not when old files change. It's possible that the dataflow activity updates an existing file rather than producing a new one when it moves data to the blob storage.

Agreed with @Joel Cochran in Blob path ends with you are passing train_data.parquet if trigger did not find any particular file with name contain similar pattern it will not trigger the pipeline.

You may tweak the trigger to look for changes in both new and current files to fix this. This may be achieved by including the only .parquet in the Blob path ends with section of the trigger setup, which will make the trigger react to any changes to files in the supplied path.

Specify the correct details.

It's possible that the trigger is not configured to detect changes made by the dataflow activity. Check the trigger's settings to ensure that it is monitoring the correct blob container and that it is set up to detect the appropriate types of changes, such as new or modified blobs.