Kubernetes daemonset pods getting restarted for no clear reason

Question

I am deploying an application using a daemonset on a Kubernetes cluster. The application is expected to run as one instance per node on the clusters. The application pod consists of two containers - one is the actual application and the other is a helper proxy container.

During stability testing, I am noticing that the application pod on all the nodes of the cluster is getting restarted at approximately the same time. I am trying to understand what could be the potential reasons for such a restart across the cluster.

Things I have checked so far:

• There don't seem to be any eviction-related issues.
• No cluster node reboots or restarts.
• It doesn't seem like we are hitting any resource limits, but I can't say with 100% certainty. However, I don't see anything relevant in the logs.
• Nothing interesting in either syslog, kubectl events, or other logs.
• I am seeing a lot of logs around the Garbage Collector trying to free space but failing. However, from what I understand, GC shouldn't impact running containers.

My question is:

• What can I do to debug this further? I have already enabled verbose logging for kubectl to the max but couldn't find anything. I am just seeing the following DELETE call, but it is not clear why the call is being made.

Application Yaml File

apiVersion: apps/v1
kind: DaemonSet 
metadata:
  name: test-app
  labels:
    app: test-app
spec:
  selector:
    matchLabels:
      app: test-app
  template:
    metadata:
        labels:
          app: test-app
    spec:
      containers:
        - name: test-app
          image: test-app:0.0.1 
          imagePullPolicy: IfNotPresent
          resources:
            requests:
              ephemeral-storage: "100Mi"
              memory: "1Gi"
            limits:
              ephemeral-storage: "100Mi"
              memory: "1Gi"
          ports:
            - containerPort: 12345
              hostPort: 12345
          volumeMounts:
            - name: log-volume
              mountPath: /var/log/
        - image: proxy:0.0.1
          imagePullPolicy: IfNotPresent
          name: proxy
          resources:
            requests:
              cpu: 10m
              memory: 10m
      volumes:
        - name: log-volume
          hostPath:
            path: /var/log/

DELETE Logs seen in kubelet

kubelet 284030  I0522 17:09:07.287913  284030 config.go:293] "Setting pods for source" source="api"
kubelet 284030  I0522 17:09:07.301398  284030 kubelet.go:2405] "SyncLoop DELETE" source="api" pods=["default/test-app-abcde"]
kubelet 284030  I0522 17:09:07.303019  284030 pod_workers.go:856] "Pod is marked for graceful deletion, begin teardown" pod="default/test-app-abcde" podUID="123456-12345-123456-12345" updateType="update"

As mentioned above, I am NOT expecting pods to restart and trying to understand how can I debug this further to understand the root-cause for restarts.

Answer 1

Look at the last entry in the above given logs.

It says, 'Pod is marked for graceful deletion'. You need to check kubelet logs at that time why the pod was marked for deletion.

Including readiness and liveness probes to the deployment manifest might help.