kz28
kz28

Reputation: 803

Docker stats command shows zero CPU utilization for some containers but normal for others on the same node

I am running an Openwhisk cluster with two Ubuntu servers as invokers. While running some workload, some containers on one of my invoker (invoker0) reports 0.00% cpu utilization (by docker stats) even though there are intensive load on the two containers (pinned to specific CPU cores by --cpuset-cpus argument). However, the openwhisk system generated container looks showing cpu utilization correctly.

CONTAINER ID   NAME                                CPU %     MEM USAGE / LIMIT    MEM %     NET I/O          BLOCK I/O    PIDS
c19df0752f01   wsk00_8_warm_python3_func1_pin_0    0.00%     6.797MiB / 256MiB    2.66%     2.94kB / 476B    0B / 0B      5
70914e16d4d2   wsk00_7_warm_python3_func2_pin_01   0.00%     7.316MiB / 256MiB    2.86%     2.95kB / 476B    0B / 0B      6
14d8ce8a35a9   invoker0                            4.80%     210.6MiB / 62.6GiB   0.33%     7.3MB / 29.8MB   0B / 119kB   74

As shown above, the first, second container always show 0.00%, the 3rd looks normal. This problem does NOT happen on my other invoker server (invoker1). The docker info output on the problematic (invoker0) server is:

Client: Docker Engine - Community
 Version:    24.0.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.20.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 3
  Running: 1
  Paused: 2
  Stopped: 0
 Images: 91
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 5.4.0-148-generic
 Operating System: Ubuntu 20.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 62.6GiB
 Name: panic-cloud-xe3nv-03.cs.rutgers.edu
 ID: b61367f3-f91a-40d3-8924-159ae7c87d4f
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

The docker info on my normal server (invoker1) is :

Client: Docker Engine - Community
 Version:    24.0.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.20.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 2
  Running: 1
  Paused: 0
  Stopped: 1
 Images: 83
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 5.4.0-137-generic
 Operating System: Ubuntu 20.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 62.48GiB
 Name: panic-cloud-xs-06.cs.rutgers.edu
 ID: a7a9e264-1483-48bc-a9b2-c02f20f5ce44
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

I feel this is like some docker or Linux system config issue. It shouldn't be a issue with Openwhisk. Could anyone help me figure out the reason? Thank you!

PS: When I start a docker container outside the openwhisk runtime, the utilization reading is normal.

Upvotes: 1

Views: 301

Answers (0)

Related Questions