Azure Data Factory Storage Event Trigger File Path

Question

I have the below scenario:

• There will be files coming to ADLS Gen 2 at regular intervals. There is no schedule. The folder structure looks like this: f1/sf1/sf2/sf3/file(s).
• The Data Factory trigger should trigger as soon as any files are available in sf3
• After the Data Factory is triggered, we have to pickup the file(s) path along with the file(s) name and pass it as an input to an ADB notebook

Any suggestion on the approach please.

Answer 1

Check out the cloudfiles streaming source in Databricks. See Streaming data processing and Databricks Autoloader. You can run the job continuously, or just once for each new file.

It will track the files for you, can run continuously or be triggered by Data Factory, but you don't need Data Factory to pass the filename(s) to Databricks.

Picking up the filename from the trigger event is also supported, and you can pass those as parameters to databricks.