CODEWITHSUNDEEP
azurebusiness-intelligenceazure-data-factoryazure-data-catalog
Gaurav Sharma
Gaurav Sharma

Reputation: 586

Ingestion -> Transform -> Process -> Stage Tools in Azure

I don't work with database so want to confirm best practices before I start. I have been given a project to where I am supposed to take few differently modeled data files, process them and show some standard reporting.

It will be batch ingestion, can be weekly or daily max. The cloud we will be using is Microsoft Azure. I am thinking of below tools for the job from my development experience. But will love some advise from you guys:

-> Ingestion [and stage] => write a rest api which takes the data and stage it in Elastic

-> Processing -> Azure HDInsight.. I haven't used it but from brief reading, that sounds like the recommended choice.

-> Staging and Reporting -> back to Elastic search, with Kibana/Graphana. I have used these tools so preferring it.

I also read some articles recommending Azure Data Factory. Look forward to do some suggestion.

Regards, Gaurav Sharma

Upvotes: 0

Views: 194

Answers (1)

Martin Esteban Zurita
Martin Esteban Zurita

Reputation: 3209

For the ingestion you can use Data Factory, it supports calling a rest api with a WebActivity. https://learn.microsoft.com/en-us/azure/data-factory/control-flow-web-activity

For processing, have you considered storing the data in a Data Lake Store and using U-SQL to make the transformations? I'm just mentioning it because you may have skipped considering it. https://learn.microsoft.com/en-us/azure/data-lake-analytics/

For reporting Microsoft offers Power Bi, it can connect to a lot of sources included rest APIs. It may or may not be useful for you, depending on the requirements of your project. https://app.powerbi.com

Hope this helps!

Upvotes: 1

Related Questions