CODEWITHSUNDEEP
azureazure-webjobsazure-functionsazure-data-lake
AnshuK
AnshuK

Reputation: 55

Azure Data Lake Store and Azure SQL with WebJob/Azure Function

I need to upload WEB API response files into Azure Data Lake. Then I have to dump those files into Azure SQL tables.

Above both processes must be scheduled to execute on hourly basis. Should I use Azure Web Jobs or Azure Function.

Upvotes: 4

Views: 1006

Answers (2)

Amit Kulkarni
Amit Kulkarni

Reputation: 965

Azure Data Factory is probably the better mechanism to drive this recurring hourly pipeline. More details here.

https://learn.microsoft.com/en-us/azure/data-factory/data-factory-scheduling-and-execution

Upvotes: 3

Mikhail Shilkov
Mikhail Shilkov

Reputation: 35124

If you are running Azure Functions on Consumption plan, the function call must complete within 5 minutes, which might be not enough for big data sets.

For the rest Functions and Web Jobs are similar for your scenario. Functions are actually running on top of Web Jobs. And if you don't need any advanced features of Functions (e.g. bindings), I would go for a Job.

Upvotes: 2

Related Questions