CODEWITHSUNDEEP
dockerkubernetessystemd-journalddocker-logs
Abhay
Abhay

Reputation: 149

kubernetes pod logging broken with journald logging driver

Looks like journald pod logging ('kubectl logs' command) doesn't work with kubernetes v1.18.x. Is there a way to make this work with v1.18.x? I've set up a multinode cluster with docker logging driver as 'journald' (using /etc/docker/daemon.json) and using systemd-journal-gatewayd with pull approach to aggregate historical logs. However, I'm very much interested in being able to tail current logs using 'kubectl logs' or, 'kubectl logs -l 'app=label' --prefix -f' (for cluster-wide logs).

# kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

# docker version
Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Thu Apr 26 07:20:16 2018
 OS/Arch:      linux/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Thu Apr 26 07:23:58 2018
  OS/Arch:      linux/amd64
  Experimental: false

# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"

Running 'kubectl logs' gives the following error:

# kubectl logs <pod-name>
failed to try resolving symlinks in path "/var/log/pods/default_<pod-name>_xxxx/<container-name>/0.log": lstat /var/log/pods/default_<pod-name>_xxxx/<container-name>/0.log: no such file or directory

Upvotes: 5

Views: 5421

Answers (2)

SpriteCZ
SpriteCZ

Reputation: 31

In file /etc/docker/daemon.json change "log-driver": "journald" to "log-driver": "json-file". And do it on all nodes.

Upvotes: 0

kool
kool

Reputation: 3613

After you changed logging driver, the location of logs has changed to host's journal. So far the logs can be retrieved by docker but you have to let kubelet know about the changes.

You can do it by passing --log-dir=/var/log to kubelet. After adding the flag you have run systemctl daemon-reload and restart kubelet. It has to be done on all of the nodes.

Upvotes: 3

Related Questions