CODEWITHSUNDEEP
kubernetesmonitoringprometheus
Anurag Bhargav
Anurag Bhargav

Reputation: 21

How to get the logs of an application from one container into another container in kubernetes

We are planning to implement Kubernetes application monitoring using Prometheus. Our application is running in apache and deployed to kubernetes cluster. We are developing Custom Apache exporter which will run in same POD as my application container & gather metrics from apache and push them to Prometheus. As part of this monitoring we need to parse access.log in apache to get number of 2xx, 3xx,4xx & 5xx requests. But Our application that is deployed in Kubernetes doesnt store any logs inside the POD, but it logs to STDOUT. So I can access logs from kubectl logs command. Now my Apache exporter that I am building is hosted in same POD as my application container. Now my question is How can i get application logs (access.log) to be present/accessible to my apache exporter container, so that I can parse the logic and comeup with metrics like number of 2xx,3xx,4xx & 5xx.

Upvotes: 2

Views: 2645

Answers (2)

hariK
hariK

Reputation: 3059

Kubernetes stores container's STDOUTs at /var/logs/containers and /var/logs/pods of each node in a format POD_NAMESPACE_CONTAINER_CONTAINERID.log and NAMESPACE_POD_CONTAINERID/CONTAINER/*.log respectively. These files will be deleted once the pod is evitced from the node.

You can mount either of them to your exporter pod to read logs. Problems are,

  • CONTAINER ID is not available as an ENV in Kubernetes. We can't generate the file name and mount exactly the file that we want. We have to mount the entire /var/log/containers/ or /var/log/pods directory.
  • Mounting a writable hostPath is highly vulnerable in Kubernetes. So, make sure to set it as readOnly file system or configure PodSecurityPolicy.

YAML

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: nginx
  name: nginx
spec:
  containers:
  - image: nginx
    name: nginx
    resources: {}
  - image: busybox
    name: exporter
    command: ["cat"]
    tty: true
    volumeMounts:
    - mountPath: /var/log/
      name: logs
      readOnly: true
  volumes:
  - name: logs
    hostPath:
      path: /var/log/containers/
      # path: /var/log/pods/
      type: Directory
  dnsPolicy: ClusterFirst
  restartPolicy: Never
status: {}

Logs

$ kubectl exec -it nginx -n dev -c exporter sh -- ls /var/log | grep nginx
nginx_dev_exporter-41a2a80de1b899d1ee848378d2bf31285658bf86700671e560588ac69b2717b4.log
nginx_dev_nginx-e4c12df0c9dae129ed113f4195d750b447a570a686e3b5608e1af37ced788549.log

Upvotes: 4

paltaa
paltaa

Reputation: 3244

Use a log aggregation tool, like loki for example, which is well integrated with kubernetes and grafana, then you can do all this parsing in a grafana dashboard and create the alarms there

Upvotes: 0

Related Questions