CODEWITHSUNDEEP
systemdsystemd-journald
r.t
r.t

Reputation: 11

journalctl stops logging systemd service logs with systemd_unit field

This is a question about journalctl and how systemd log entries are produced. My OS is RHEL8.

I've got a systemd service setup to run kubelet as a systemd service, and I would like to use journalctl to tail the logs using the unit flag with "journalctl -u kubelet".

When I initially start the systemd service, I can see kubelet logs show up in /var/log/messages and I can also filter them using "journalctl -u kubelet". However, very shortly after the service is started, journalctl goes quiet when filtering with "-u kubelet", yet kubelet logs continue to be dumped into /var/log/messages. If I filter journalctl with "journalctl --identifier kubelet" instead of the "-u kubelet" I do see all the logs that are in /var/log/messages.

Using "-o json-pretty", I can see that the initial logs produced by the kubelet process have journald log entries with:

    "_SYSTEMD_CGROUP" : "/system.slice/kubelet.service",
    "_SYSTEMD_INVOCATION_ID" : "b422558179854d55a44d0ea6f7240828",
    "_SYSTEMD_SLICE" : "system.slice",
    "_SYSTEMD_UNIT" : "kubelet.service",

Logs produced shortly after starting the service is started seem to drop the unit property, and look like:

    "_SYSTEMD_CGROUP" : "/systemd/system.slice",
    "_SYSTEMD_INVOCATION_ID" : "b422558179854d55a44d0ea6f7240828",
    "_SYSTEMD_SLICE" : "-.slice",

I think the fact that the logs start being produced without the "_SYSTEMD_UNIT" property indicates why filtering them with "-u" stops working, but I'd like to know why my service initially starts producing logs with the unit property, and then stops. Any clues would be appreciated.

Upvotes: 0

Views: 859

Answers (1)

r.t
r.t

Reputation: 11

Turns out this had more to do with kubelet configuration than it did journald configuration. kubelet needed to designate the correct cgroup in the kubeletCgroups config.

Upvotes: 0

Related Questions