Reputation: 803
I am running an Openwhisk cluster with two Ubuntu servers as invokers. While running some workload, some containers on one of my invoker (invoker0) reports 0.00% cpu utilization (by docker stats
) even though there are intensive load on the two containers (pinned to specific CPU cores by --cpuset-cpus
argument). However, the openwhisk system generated container looks showing cpu utilization correctly.
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
c19df0752f01 wsk00_8_warm_python3_func1_pin_0 0.00% 6.797MiB / 256MiB 2.66% 2.94kB / 476B 0B / 0B 5
70914e16d4d2 wsk00_7_warm_python3_func2_pin_01 0.00% 7.316MiB / 256MiB 2.86% 2.95kB / 476B 0B / 0B 6
14d8ce8a35a9 invoker0 4.80% 210.6MiB / 62.6GiB 0.33% 7.3MB / 29.8MB 0B / 119kB 74
As shown above, the first, second container always show 0.00%, the 3rd looks normal. This problem does NOT happen on my other invoker server (invoker1). The docker info output on the problematic (invoker0) server is:
Client: Docker Engine - Community
Version: 24.0.5
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.11.2
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.20.2
Path: /usr/libexec/docker/cli-plugins/docker-compose
scan: Docker Scan (Docker Inc.)
Version: v0.23.0
Path: /usr/libexec/docker/cli-plugins/docker-scan
Server:
Containers: 3
Running: 1
Paused: 2
Stopped: 0
Images: 91
Server Version: 24.0.5
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
runc version: v1.1.8-0-g82f18fe
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
Kernel Version: 5.4.0-148-generic
Operating System: Ubuntu 20.04.6 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 62.6GiB
Name: panic-cloud-xe3nv-03.cs.rutgers.edu
ID: b61367f3-f91a-40d3-8924-159ae7c87d4f
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
The docker info on my normal server (invoker1) is :
Client: Docker Engine - Community
Version: 24.0.5
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.11.2
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.20.2
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 2
Running: 1
Paused: 0
Stopped: 1
Images: 83
Server Version: 24.0.5
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
runc version: v1.1.8-0-g82f18fe
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
Kernel Version: 5.4.0-137-generic
Operating System: Ubuntu 20.04.6 LTS
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 62.48GiB
Name: panic-cloud-xs-06.cs.rutgers.edu
ID: a7a9e264-1483-48bc-a9b2-c02f20f5ce44
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
I feel this is like some docker or Linux system config issue. It shouldn't be a issue with Openwhisk. Could anyone help me figure out the reason? Thank you!
PS: When I start a docker container outside the openwhisk runtime, the utilization reading is normal.
Upvotes: 1
Views: 301