CODEWITHSUNDEEP
azureazure-storageazure-data-factory
gabi
gabi

Reputation: 1376

Azure: Run a data factory activity when a new file is added to data lake store

I have a large dataset on Azure Data lake store and a few files might be added/updated there daily. How can I process these new files without reading the entire dataset each time? I need to copy these new files using Data Factory V1 to SQL server.

Upvotes: 0

Views: 335

Answers (1)

Fang Liu
Fang Liu

Reputation: 2363

If you could use ADF V2, then you could use get metadata activity to get the lastModifiedDate Properties of each file and then only copy new files. You could reference this doc. https://learn.microsoft.com/en-us/azure/data-factory/control-flow-get-metadata-activity

Upvotes: 2

Related Questions