CODEWITHSUNDEEP
kuberneteslogginggoogle-kubernetes-engine
Tho Quach
Tho Quach

Reputation: 1415

Kubernetes logs from kubectl logs command is different to /var/log/containers in GKE cluster

I have created a GKE cluster on GCP.

Kubernetes logs from kubectl logs command is different to /var/log/containers

kubectl

{"method":"GET","path":"/healthz","format":"*/*","controller":"Public::PublicPagesController","action":"healthz","status":204,"duration":0.39,"view":0.0,"request_id":"ca29b519-d1e8-49a2-95ae-e5f23b60c36f","params":{},"custom":null,"request_time":"2022-04-27T15:25:43.780+00:00","process_id":6,"@version":"vcam-backend-vvcam-72_shareholder_event-rc16","@timestamp":"2022-04-27T15:25:43.780Z","message":"[204] GET /healthz (Public::PublicPagesController#healthz)"}

And logs in /var/log/containers, something add timestamp into the beginning of my container logs:

2022-04-27T15:25:43.780523421Z stdout F {"method":"GET","path":"/healthz","format":"*/*","controller":"Public::PublicPagesController","action":"healthz","status":204,"duration":0.39,"view":0.0,"request_id":"ca29b519-d1e8-49a2-95ae-e5f23b60c36f","params":{},"custom":null,"request_time":"2022-04-27T15:25:43.780+00:00","process_id":6,"@version":"vcam-backend-vvcam-72_shareholder_event-rc16","@timestamp":"2022-04-27T15:25:43.780Z","message":"[204] GET /healthz (Public::PublicPagesController#healthz)"}

I want my application log will be consistent, I want it in json format like logs from kubectl command, so I can parse and analyze more.

I want to remove this part: 2022-04-27T15:25:43.780523421Z stdout F

Does anybody meet this problem? How can I make containers logs same as kubectl command logs?

GKE Version:

Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.10-gke.2000", GitCommit:"0823380786b063c3f71d5e7c76826a972e30550d", GitTreeState:"clean", BuildDate:"2022-03-17T09:22:22Z", GoVersion:"go1.16.14b7", Compiler:"gc", Platform:"linux/amd64"}

Docker daemon.json

{
  "pidfile": "/var/run/docker.pid",
  "iptables": false,
  "ip-masq": false,
  "log-level": "warn",
  "bip": "169.254.123.1/24",
  "mtu": 1460,
  "storage-driver": "overlay2",
  "live-restore": true,
  "log-driver": "json-file",
  "log-opts": {
      "max-size": "10m",
      "max-file": "5"
    }
}

Notes: I notice that the timestamp in the beginning of log line only show when we add option docker logs -t, docs here But I still do not know how to fix this problem in GKE cluster.

Upvotes: 5

Views: 2231

Answers (1)

Tho Quach
Tho Quach

Reputation: 1415

This problem is related to Container Runtime Interface (CRI). You can read about CRI here.

For now, I'm still can not change the log format as I want, I just adapt to this new format. This format is called CRI log format, the CRI default of GKE cluster always produce log in this format, and aggregation log applications adapt to this new CRI log format too:

  • This is PR of Grafana/Loki to support CRI log format: PR
  • Fluent-bit create a new parser for CRI log format: Docs

So I think you need to change the way you approach this problem: If we can not change the log format as we want, we can use application support this log format.

Notes: I'm not sure but I think this problem come from: Kubernetes is removing support for Docker as a container runtime - docs , and the new container runtime produce this CRI log format.

Upvotes: 4

Related Questions