CODEWITHSUNDEEP
jsondockerloggingdebiandaemon
Mysteryos
Mysteryos

Reputation: 5791

Docker daemon fails to process logs

Docker daemon is unable to parse the json log and throws an unexpected EOF error. We are investigating the root cause of the issue.

Environment:

Docker daemon log:

Apr 29 14:31:05 Prod-IS dockerd[30810]: time="2020-04-29T14:31:05.070677515+02:00" level=warning msg="got error while decoding json" error="unexpected EOF" retries=345
Apr 29 14:31:05 Prod-IS dockerd[30810]: time="2020-04-29T14:31:05.070695689+02:00" level=warning msg="got error while decoding json" error="unexpected EOF" retries=346
Apr 29 14:31:05 Prod-IS dockerd[30810]: time="2020-04-29T14:31:05.070712630+02:00" level=warning msg="got error while decoding json" error="unexpected EOF" retries=347
Apr 29 14:31:05 Prod-IS dockerd[30810]: time="2020-04-29T14:31:05.070732299+02:00" level=warning msg="got error while decoding json" error="unexpected EOF" retries=348
Apr 29 14:31:05 Prod-IS dockerd[30810]: time="2020-04-29T14:31:05.070755016+02:00" level=warning msg="got error while decoding json" error="unexpected EOF" retries=349
Apr 29 14:31:05 Prod-IS dockerd[30810]: time="2020-04-29T14:31:05.070773699+02:00" level=warning msg="got error while decoding json" error="unexpected EOF" retries=350

Docker daemon configuration

"log-driver": "json-file",
"log-opts": {
  "mode": "non-blocking",
  "max-size": "500m",
  "max-file": "3"
}

Storage space is available on partition where docker writes its logs.

Your insight in the matter is most welcome.

Upvotes: 15

Views: 15399

Answers (2)

Mysteryos
Mysteryos

Reputation: 5791

The aforementioned issue will cause an abnormal spike in CPU usage of the docker (version <= 19) process and constantly stay at that level until you fix the logs.

To fix the logs & cpu usage, you've to truncate all the container logs (since we are unable to determine which container has faulty logs). It works only if your logs are in json format:

truncate -s 0 /var/lib/docker/containers/*/*-json.log

Please see Andre's answer for a permanent fix.

Upvotes: 12

Andre Leon Rangel
Andre Leon Rangel

Reputation: 1799

Issue=Docker Engine Error json-file: fix sporadic unexpected EOF errors

I had spent some time researching and trying to find errors in the docker logs.

There is a bug in docker engine 20.10.5 which is the version running in docker. A Github PR fixes this. https://github.com/moby/moby/pull/42104

I found the error while running

$ journalctl -u docker.service
level=warning msg="got error while decoding json" error="unexpected EOF" retries=19999

Docker driver for logs has a bug and it wont handle some log formatting. The bug then floods the system's log with retries messages. This situation generates a very annoying CPU load.

Docker engine latest is 20.10.7. The issue has been fixed in release Docker engine 20.10.6 https://docs.docker.com/engine/release-notes/#20106

Upvotes: 5

Related Questions