CODEWITHSUNDEEP
azure-aks
neManiac
neManiac

Reputation: 355

How can I query CPU and memory usage per container in AKS?

AKS Insights

So, I can plainly see per container stats in the insights tab of AKS. These must come from somewhere, but I can only find per node stats when querying logs/metrics. How can I query this (in order to build a workbook).

Upvotes: 10

Views: 15362

Answers (1)

Patrick Healy
Patrick Healy

Reputation: 196

That data is in the Perf table in the LogManagement section: screenshot of log analytics query showing raw performance data

The documentation page on How to query logs from Azure Monitor for containers has example queries you can start with: screenshot of documentation page showing sample query

Querying this data takes a bit of parsing, because the Computer field always shows the name of the node the data was gathered from, not the pod. In order to get pod/container specific data, you have to look at records with ObjectName == 'K8SContainer' and parse the InstanceName field, which contains the data you need. InstanceName is built like this: /subscriptions/SUBSCRIPTIONID/resourceGroups/RESOURCEGROUPNAME/providers/Microsoft.ContainerService/managedClusters/CLUSTERNAME/PODUID/CONTAINERNAME. Given that data, we can parse out the PodUid and join with KubePodInventory to get the identifying information for that Pod.

Here's an example query:

Perf
| where ObjectName == 'K8SContainer' and TimeGenerated > ago(1m)
| extend PodUid = tostring(split(InstanceName, '/', 9)[0]), Container = tostring(split(InstanceName, '/', 10)[0])
| join kind=leftouter (KubePodInventory | summarize arg_max(TimeGenerated, *) by PodUid) on PodUid
| project TimeGenerated, ClusterName, Namespace, Pod = Name, Container, PodIp, Node = Computer, CounterName, CounterValue

This query produces a result like this, which should contain the data you need: performance query results table

As a side note - the Computer field always shows the node name because that's where the OMS agent is running. It's gathering statistics at the node level, but those stats include the memory and CPU usage for each cgroup, which is the backing CPU/memory isolation and limiting technology behind containers in general, just like how namespaces are used to separate networking, filesystems, and process IDs.

Upvotes: 10

Related Questions