CODEWITHSUNDEEP
azureazure-data-factory
naveen kumar
naveen kumar

Reputation: 188

Is it possible to Monitor Azure Integration Runtime?

I am running few Data Pipelines in Azure Data Factory and its using Azure Integration Runtime for the compute. I am trying to Monitor the CPU/Memory Usage Pipelines Consume and Utilise Azure IR. I have checked in the Azure Monitor but the CPU / Memory Metrics are for Self Hosted Integration Runtime I think. Also, with the Diagnostic Setting Enabled, I tried to verify the details in the Logs too but these details are not available. Can anyone help to know more options?

Upvotes: 1

Views: 2945

Answers (2)

Trent Tamura
Trent Tamura

Reputation: 1145

If you are referring to the Azure AutoResolveIntegrationRuntime, then no there is not, and this is why (from https://www.cathrinewilhelmsen.net/integration-runtimes-azure-data-factory/)

Screenshot on Azure Integration Runtime

Microsoft has massive elastic pools across the various locations/regions they offer Azure, and at runtime ADF determines what pool/hardware it will use to perform the Pipeline activities. So there is really no way (and no need) to monitor the Azure Autoresolve IR. But if you are interested in monitoring Self-Hosted IR's then there are many ways to do it.

One simple and straight forward way to do it is by creating Azure Dashboards in the Metrics portion of Azure Monitor. As you can see from the screenshot below it provides good visual representation of usage/resources over time. Azure Dashboard Example As you can see I'm visualizing the integration Runtime itself (CPU/Memory) as well as the Azure VM that is hosting the Integration Runtime. On top of this you can go into the Metrics dashboard to set up alerts if certain conditions are met (eg AVG CPU % usage is over 75% for the last 15 minutes). These alerts can send you a text message, or email... and even do things as complicated as triggering a LogicApp or WebHook for automated scaling up/out, advanced notifications, etc.

This in my opinion is the best way to monitor but another option could be to call the Azure Data Factory REST API to get monitor data for the Integration Runtimes

POST https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/integrationRuntimes/{integrationRuntimeName}/monitoringData?api-version=2018-06-01

But this method would require you to incrementally pull in data, store it, parse it, and then visualize it or act upon it when that is already very well built in for you. Sometimes it's fun to recreate wheels though.

Upvotes: 2

Abhishek Khandave
Abhishek Khandave

Reputation: 3230

Yes It is possible to Monitor Azure Integration Runtime.

"Pipeline Runs" in Monitoring has the option to check the CPU Utilization specific to pipeline, Integration Runtime and more specific filters. You can find here, how its done.

Upvotes: 1

Related Questions