CODEWITHSUNDEEP
databricksazure-databricksaws-databricks
MJ029
MJ029

Reputation: 199

Azure DataBricks Cluster usage metrics

Is there any way that I can log Azure databricks cluster usage metrics like CPU, Memory, Network, Throughput usage and etc.... ?

There are ways before 13.3 series, is there any way post 13.3 series (Excluding DataDog & Grafana)

Upvotes: 0

Views: 360

Answers (1)

Axel R.
Axel R.

Reputation: 1300

You can find some of these metrics in the System Tables and more specifically in the node_timeline table.

SELECT *
FROM system.compute.node_timeline
WHERE workspace_id = "<your_workspace_id"
AND cluster_id = "<your_cluster_id>"

System Table Query Example

It is also available visually in the Metrics Tab of your cluster.

https://<workspace_url>/compute/clusters/<cluster_id>/metrics

enter image description here

Upvotes: 1

Related Questions