cpuPercent metric from docker stats vs cgroups

Question

I am new to cgroups, and trying to get the container stats using cgroups. Previously i was using docker stats but, trying to gather similar metrics with cgroups as well.

In docker stats, cpu stats section is like below:

"cpu_usage": {
        "total_usage": 27120642519,
        "percpu_usage": [27120642519],
        "usage_in_kernelmode": 4550000000,
        "usage_in_usermode": 19140000000
    },
    "system_cpu_usage": 42803030000000,

And, the cpu % metric is calculated using the below equation:

cpuDelta = float64(v.CpuStats.CpuUsage.TotalUsage - previousCPU)
systemDelta = float64(v.CpuStats.SystemUsage - previousSystem)
cpuPct = cpuDelta/systemDelta

I am looking at cgroups to gather systemUsage and the totalUsage, but it does not seem to have similar metrics:

cgroups has a pseudo file cpuacct.stats which has user and system ticks, but these are matching only with usage_in_user_mode and usage_in_kernel_mode from the docker stats output.

and cpuacct.usage_per_cpu pseudo file has a usage per cpu, which is matching with the total_usage from docker stats output above.

$cat cpuacct.stat 
user 1914
system 455

$cat cpuacct.usage_percpu 
27120642519 

But, i could not find any way to figure out how to gather "systemUsage" from cgroups.

Any leads will be of great help!

Thanks!

Answer 1

The answer to your question doesn't lies in the cgroups. Please refer the below mentioned point:

func calculateCPUPercentUnix(previousCPU, previousSystem uint64, v *types.StatsJSON) float64 {
    var (
        cpuPercent = 0.0
        // calculate the change for the cpu usage of the container in between readings
        cpuDelta = float64(v.CPUStats.CPUUsage.TotalUsage) - float64(previousCPU)
        // calculate the change for the entire system between readings
        systemDelta = float64(v.CPUStats.SystemUsage) - float64(previousSystem)
    )
    if systemDelta > 0.0 && cpuDelta > 0.0 {
        cpuPercent = (cpuDelta / systemDelta) * float64(len(v.CPUStats.CPUUsage.PercpuUsage)) * 100.0
    }
    return cpuPercent
}

1. The "system_cpu_usage" of the Docker stats API refers to the CPU usage of the host.
2. The "cpu_usage" > "total_usage" of the Docker stats API refers to the per-CPU usage of the container.
3. Hence after calculation of the (cpuDelta/systemDelta) we get the per-CPU usage per system CPU.
4. Now we need to multiply the result of the step 3 and the total number of CPU allocated to the docker container to get the total CPU usage per system CPU.
5. The result of step 4 when multiplied by 100 gives us the CPU utilization in percentage.

Back to question: How System CPU is calculated by docker?

To calculate the system CPU usage docker uses the "/proc/stat" defined by POSIX. It looks for the CPU statistics line and then sums up the first seven fields provided. The golang code written to perform the required steps is mentioned below.

https://github.com/rancher/docker/blob/3f1b16e236ad4626e02f3da4643023454d7dbb3f/daemon/stats_collector_unix.go#L137

// getSystemCPUUsage returns the host system's cpu usage in
// nanoseconds. An error is returned if the format of the underlying
// file does not match.
//
// Uses /proc/stat defined by POSIX. Looks for the cpu
// statistics line and then sums up the first seven fields
// provided. See `man 5 proc` for details on specific field
// information.
func (s *statsCollector) getSystemCPUUsage() (uint64, error) {
    var line string
    f, err := os.Open("/proc/stat")
    if err != nil {
        return 0, err
    }
    defer func() {
        s.bufReader.Reset(nil)
        f.Close()
    }()
    s.bufReader.Reset(f)
    err = nil
    for err == nil {
        line, err = s.bufReader.ReadString('\n')
        if err != nil {
            break
        }
        parts := strings.Fields(line)
        switch parts[0] {
        case "cpu":
            if len(parts) < 8 {
                return 0, derr.ErrorCodeBadCPUFields
            }
            var totalClockTicks uint64
            for _, i := range parts[1:8] {
                v, err := strconv.ParseUint(i, 10, 64)
                if err != nil {
                    return 0, derr.ErrorCodeBadCPUInt.WithArgs(i, err)
                }
                totalClockTicks += v
            }
            return (totalClockTicks * nanoSecondsPerSecond) /
                s.clockTicksPerSecond, nil
        }
    }
    return 0, derr.ErrorCodeBadStatFormat
}

Please match the "system_cpu_usage" of docker stats API with the output of below mentioned command to confirm:

cat /proc/stat|grep -w cpu|awk '{split($0,a,\" \"); sum=0; for(i=2;i<8;i++)(sum+=a[i])} END{print sum }'