CODEWITHSUNDEEP
elasticsearchkubernetesloggingfilebeat
itaied
itaied

Reputation: 7107

Access k8s pod logs generated from ssh exec

I have a filebeat configured to send my k8s cluster logs to Elasticsearch.
When I connect to the pod directly (kubectl exec -it <pod> -- sh -c bash),
the generated output logs aren't being sent to the destination.

Digging at k8s docs, I couldn't find how k8s is handling STDOUT from a running shell.

How can I configure k8s to send live shell logs?

Upvotes: 2

Views: 2367

Answers (2)

KayV
KayV

Reputation: 13845

Use the following process:

  1. Make changes in your application to push logs to STDOUT. You may configure this in your logging configuration file.
  2. Configure file to read those STDOUT logs (which eventual is some docker log file location like /var/log etc)
  3. Start your file as a DeamonSets, so that logs from new pods and nodes can be anatomically pushed to ES.
  4. For better readability of logs, make sure you push logs in json format.

Upvotes: 0

AndD
AndD

Reputation: 2701

Kubernetes has (mostly) nothing to do with this, as logging is handled by the container environment used to support Kubernetes, which is usually docker.

Depending on docker version, logs of containers could be written on json-file, journald or more, with the default being a json file. You can do a docker info | grep -i logging to check what is the Logging Driver used by docker. If the result is json-file, logs are being written down on a file in json format. If there's another value, logs are being handled in another way (and as there are various logging drivers, I suggest to check the documentation about them)

If the logs are being written on file, chances are that by using docker inspect container-id | grep -i logpath, you'll be able to see the path on the node.

Filebeat simply harvest the logs from those files and it's docker who handles the redirection between the application STDOUT inside the container and one of those files, with its driver.

Regarding exec commands not being in logs, this is an open proposal ( https://github.com/moby/moby/issues/8662 ) as not everything is redirected, just logs of the apps started by the entrypoint itself.

There's a suggested workaround which is ( https://github.com/moby/moby/issues/8662#issuecomment-277396232 )

In the mean time you can try this little hack....

echo hello > /proc/1/fd/1

Redirect your output into PID 1's (the docker container) file descriptor for STDOUT

Which works just fine but has the problem of requiring a manual redirect.

Upvotes: 4

Related Questions