B.Hong
Reputation: 89
podman - Failed to stop any containers by using "sudo podman stop"
I have encountered a problem with podman. I have successfully ran couple containers within it. But when I issue "sudo podman stop/restart xxx", I always get below error (used pgadmin4 as an example), then the container status stuck at "Stopping". To recover from it, I have to restart the computer.
I found this issue only present when run the container using sudo podman, using podman without sudo are all good.
WARN[0010] StopSignal SIGTERM failed to stop container mypgadmin4 in 10 seconds, resorting to SIGKILL
Error: given PID did not die within timeout
Below is my environment
Ubuntu 24.04
Kernel: 6.8.0-35-generic #35-Ubuntu SMP PREEMPT_DYNAMIC
Remote login through ssh
Podman is installed through Ubuntu official repos using apt install.
Podman details:
Client: Podman Engine
Version: 4.9.3
API Version: 4.9.3
Go Version: go1.22.1
Built: Thu Jan 1 00:00:00 1970
OS/Arch: linux/amd64
podman ps details
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
03a444daf9a5 docker.io/dpage/pgadmin4:latest 20 hours ago Stopping 0.0.0.0:5050->80/tcp mypgadmin4
Podman inspect details:
[
{
"Id": "03a444daf9a56b0e1d662da466ef657d3e760b83966b1b931f085b9ec4c2dee5",
"Created": "2024-06-07T08:13:52.48940452Z",
"Path": "/entrypoint.sh",
"Args": [
"/entrypoint.sh"
],
"State": {
"OciVersion": "1.1.0",
"Status": "stopping",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1354,
"ConmonPid": 1352,
"ExitCode": 0,
"Error": "container 03a444daf9a56b0e1d662da466ef657d3e760b83966b1b931f085b9ec4c2dee5 must be in Created or Stopped state to be started: container state improper",
"StartedAt": "2024-06-08T04:16:37.753214803Z",
"FinishedAt": "2024-06-08T04:13:29.582636564Z",
"Health": {
"Status": "",
"FailingStreak": 0,
"Log": null
},
"CheckpointedAt": "0001-01-01T00:00:00Z",
"RestoredAt": "0001-01-01T00:00:00Z",
"StoppedByUser": true
},
"Image": "52957d72b44aeb31109430b2b681a4867a94825c02ac7cfeca31decfb65a4e18",
"ImageDigest": "sha256:61fd25f428c155027fb2aa74b913d317af11a14f55e6135484b5e86a8840520b",
"ImageName": "docker.io/dpage/pgadmin4:latest",
"Rootfs": "",
"Pod": "",
"ResolvConfPath": "/run/containers/storage/overlay-containers/03a444daf9a56b0e1d662da466ef657d3e760b83966b1b931f085b9ec4c2dee5/userdata/resolv.conf",
"HostnamePath": "/run/containers/storage/overlay-containers/03a444daf9a56b0e1d662da466ef657d3e760b83966b1b931f085b9ec4c2dee5/userdata/hostname",
"HostsPath": "/run/containers/storage/overlay-containers/03a444daf9a56b0e1d662da466ef657d3e760b83966b1b931f085b9ec4c2dee5/userdata/hosts",
"StaticDir": "/var/lib/containers/storage/overlay-containers/03a444daf9a56b0e1d662da466ef657d3e760b83966b1b931f085b9ec4c2dee5/userdata",
"OCIConfigPath": "/var/lib/containers/storage/overlay-containers/03a444daf9a56b0e1d662da466ef657d3e760b83966b1b931f085b9ec4c2dee5/userdata/config.json",
"OCIRuntime": "crun",
"ConmonPidFile": "/run/containers/storage/overlay-containers/03a444daf9a56b0e1d662da466ef657d3e760b83966b1b931f085b9ec4c2dee5/userdata/conmon.pid",
"PidFile": "/run/containers/storage/overlay-containers/03a444daf9a56b0e1d662da466ef657d3e760b83966b1b931f085b9ec4c2dee5/userdata/pidfile",
"Name": "mypgadmin4",
"RestartCount": 0,
"Driver": "overlay",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "containers-default-0.57.4",
"EffectiveCaps": null,
"BoundingCaps": [
"CAP_CHOWN",
"CAP_DAC_OVERRIDE",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_NET_BIND_SERVICE",
"CAP_SETFCAP",
"CAP_SETGID",
"CAP_SETPCAP",
"CAP_SETUID",
"CAP_SYS_CHROOT"
],
"ExecIDs": [],
"GraphDriver": {
"Name": "overlay",
"Data": {
"LowerDir": "/var/lib/containers/storage/overlay/a5c7f871d710f522023ca5dfdad42ae08922c22a6521ef6586cfc5be868d455e/diff:/var/lib/containers/storage/overlay/dc86a1c3f885aa3c52c12d8d284601453b8351a39d389643c68ef48743e1d1cf/diff:/var/lib/containers/storage/overlay/9b97c28998b2e858ec85ca6bd90bd5bac65796ec9ba935cfc207e5b2a6cdbd8f/diff:/var/lib/containers/storage/overlay/449060aaadfe49aa1bff299b09d0d7cfed4ee0613aac066a6a6c5cc31297b422/diff:/var/lib/containers/storage/overlay/ee986652d35f7ce871f3ff7ec6bccfbaf9d2cca2fbd84d2f4ec9dc985a69d1fc/diff:/var/lib/containers/storage/overlay/17bb2f5a79fd2fa0d6392b0c285632f551cdba15f00d7fd8370016fbf29ed38c/diff:/var/lib/containers/storage/overlay/6874fb546f2470531ac32bfeb5cb02e559ed7ed0ee8948e01f40d3f5bc90cb34/diff:/var/lib/containers/storage/overlay/bfcef14a6c361df75f0788cf35434d68d59683667538048dc4563146bd599aa6/diff:/var/lib/containers/storage/overlay/2a1fd15f5b430cbeaaaa09526b890da406b895ac1dbb7507d4f17bb84500ca6d/diff:/var/lib/containers/storage/overlay/e1f4108a9613a1b6d2a06b76d96166e8173849d406c0293c453459801c17e5f0/diff:/var/lib/containers/storage/overlay/a8581de914bb2816ffe235a6b98aa3761d188cbba6b7e0b45d20e7329d8d2f73/diff:/var/lib/containers/storage/overlay/74f98c324854f69f351d8c3c17822ca47b473d63f268c4bb8d2ed1d1dc966ded/diff:/var/lib/containers/storage/overlay/d4fa294b332976ab868c91a9b2115e89138635f3439dc0255a7bcfeae3fc67f2/diff:/var/lib/containers/storage/overlay/9c212d5aefe774f9ed137f7a5e497dd71046ef0f80a7bae994e0857068879fd3/diff:/var/lib/containers/storage/overlay/a6e7ea5b0f1081955cb735c56d1d4ef51c516ec66ce897b063957d1a9218cc0e/diff:/var/lib/containers/storage/overlay/02f2bcb26af5ea6d185dcf509dc795746d907ae10c53918b6944ac85447a0c72/diff",
"MergedDir": "/var/lib/containers/storage/overlay/f5378473fddf6cce0f561698ecc0d35ff6cb2df7762bc5b16d93811671157310/merged",
"UpperDir": "/var/lib/containers/storage/overlay/f5378473fddf6cce0f561698ecc0d35ff6cb2df7762bc5b16d93811671157310/diff",
"WorkDir": "/var/lib/containers/storage/overlay/f5378473fddf6cce0f561698ecc0d35ff6cb2df7762bc5b16d93811671157310/work"
}
},
"Mounts": [
{
"Type": "volume",
"Name": "fcc5478dcd526de788efa231bae5f93c9466e9ae40b1ce98af6930c11d1b0941",
"Source": "/var/lib/containers/storage/volumes/fcc5478dcd526de788efa231bae5f93c9466e9ae40b1ce98af6930c11d1b0941/_data",
"Destination": "/var/lib/pgadmin",
"Driver": "local",
"Mode": "",
"Options": [
"nodev",
"exec",
"nosuid",
"rbind"
],
"RW": true,
"Propagation": "rprivate"
}
],
"Dependencies": [],
"NetworkSettings": {
"EndpointID": "",
"Gateway": "10.88.0.1",
"IPAddress": "10.88.0.4",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "da:52:dd:8b:88:dc",
"Bridge": "",
"SandboxID": "",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"443/tcp": null,
"80/tcp": [
{
"HostIp": "",
"HostPort": "5050"
}
]
},
"SandboxKey": "/run/netns/netns-e16d008f-10f1-81b1-8f01-851badc15f3a",
"Networks": {
"podman": {
"EndpointID": "",
"Gateway": "10.88.0.1",
"IPAddress": "10.88.0.4",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "da:52:dd:8b:88:dc",
"NetworkID": "podman",
"DriverOpts": null,
"IPAMConfig": null,
"Links": null,
"Aliases": [
"03a444daf9a5"
]
}
}
},
"Namespace": "",
"IsInfra": false,
"IsService": false,
"KubeExitCodePropagation": "invalid",
"lockNumber": 6,
"Config": {
"Hostname": "03a444daf9a5",
"Domainname": "",
"User": "pgadmin",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"container=podman",
"PYTHONPATH=/pgadmin4",
"[email protected]",
"PGADMIN_DEFAULT_PASSWORD=xxxx",
"HOME=/home/pgadmin",
"HOSTNAME=03a444daf9a5"
],
"Cmd": null,
"Image": "docker.io/dpage/pgadmin4:latest",
"Volumes": null,
"WorkingDir": "/pgadmin4",
"Entrypoint": "/entrypoint.sh",
"OnBuild": null,
"Labels": null,
"Annotations": {
"io.container.manager": "libpod",
"org.opencontainers.image.stopSignal": "15"
},
"StopSignal": 15,
"HealthcheckOnFailureAction": "none",
"CreateCommand": [
"podman",
"run",
"--name",
"mypgadmin4",
"-p",
"5050:80",
"-e",
"[email protected]",
"-e",
"PGADMIN_DEFAULT_PASSWORD=xxxx",
"-d",
"docker.io/dpage/pgadmin4:latest"
],
"Umask": "0022",
"Timeout": 0,
"StopTimeout": 10,
"Passwd": true,
"sdNotifyMode": "container"
},
"HostConfig": {
"Binds": [
"fcc5478dcd526de788efa231bae5f93c9466e9ae40b1ce98af6930c11d1b0941:/var/lib/pgadmin:rprivate,rw,nodev,exec,nosuid,rbind"
],
"CgroupManager": "systemd",
"CgroupMode": "private",
"ContainerIDFile": "",
"LogConfig": {
"Type": "journald",
"Config": null,
"Path": "",
"Tag": "",
"Size": "0B"
},
"NetworkMode": "bridge",
"PortBindings": {
"80/tcp": [
{
"HostIp": "",
"HostPort": "5050"
}
]
},
"RestartPolicy": {
"Name": "",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": [],
"CapDrop": [],
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": [],
"GroupAdd": [],
"IpcMode": "shareable",
"Cgroup": "",
"Cgroups": "default",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "private",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [],
"Tmpfs": {},
"UTSMode": "private",
"UsernsMode": "",
"ShmSize": 65536000,
"Runtime": "oci",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": 0,
"OomKillDisable": false,
"PidsLimit": 2048,
"Ulimits": [
{
"Name": "RLIMIT_NPROC",
"Soft": 4194304,
"Hard": 4194304
}
],
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"CgroupConf": null
}
}
]
Can I ask what caused this issue and how can I fix it please?
Thanks in advance!
Upvotes: 1
Views: 2203
Answers (2)
mlongphe
Reputation: 31
Just recently had this issue. As to why this happened I am not sure what caused this problem.
What I did to fix it:
The error says: "Error: given PID did not die within timeout" so I figured the PID wasn't being stopped and I needed to manually stop it. What worked for me was:
podman container inspect <your-container> | grep "\"PID\":"
Then grab the PID value and kill it manually:
kill -9 <PID>
Here is a one liner for what I mentioned above:
kill -9 $(podman container inspect <your-container> -f '{{.State.Pid}}')
Edited: here is a cleaner way to do the one liner, thanks to Erik in the comments
Upvotes: 3
DerSkythe
Reputation: 525
In config section I see:
"StopSignal": 15,
Try to change to 9. Also, this command is very useful:
podman container kill -fa
Check and setup containers.conf for these values:
- stop_timeout
- exit_command_delay
Upvotes: 0
Related Questions
- podman, running a c++ application
- Get Name of Server running Podman Containers
- Assigning specific IP address to Containers using Podman
- Podman rootless containers communication issue
- How does podman behave when using sudo vs not using sudo
- Podman stop command removes container
- podman cannot see all running containers
- Podman: Changes made to podman config.json do not persist on container start