CODEWITHSUNDEEP
kubernetes
tse
tse

Reputation: 6079

How to monitor executing of `preStop` command?

I'm trying to use pod's lifecycle event. Problem is that command from preStop doesn't run at all. Is there any way to monitor if it was started? Log of the container is empty. 

      lifecycle:
        preStop:
          exec:
            command: [ "/bin/sh", "-c", "/clean.sh" ]

Upvotes: 19

Views: 14055

Answers (4)

Anish Shah
Anish Shah

Reputation: 8169

I found the logs of preStop hook failure in kubelet logs. I'm not sure if it is the default setting to show preStop hook log or need to increase the verbosity of the kubelet log. FWIW, my kubelet is configured with --v=2 verbosity.

Upvotes: 0

Adiii
Adiii

Reputation: 59976

I was looking for something, so I added some logging that help to see the logs of the script in the pod's stdout/stderr logs.

so for me, this approach help me

  • To write logs to the centralized logging system (help me to check logs in datadog)
  • Verify the script is executed properly
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "/clean.sh > /proc/1/fd/1"]

and was able to verify the logs

kubectl get pods

kubectl logs -f my_stohook_pod

/proc/PID/fd/1 will help us to redirect script logs stdout/stderr of the container main process.

Upvotes: 10

David Thomas
David Thomas

Reputation: 4396

I just want to add for the preStop hook, the pod may be terminated and not available to describe.

Another way to see the preStop error log is via kubectl events:

kubectl get events | grep FailedPreStopHook

Example:

kubectl get events | grep FailedPreStopHook                                    
5m33s       Warning   FailedPreStopHook   pod/pod-name-59988c4675-79q4p                              
Exec lifecycle hook ([/bin/kill -s SIGQUIT 1]) for Container "container_name" in Pod "pod-name-59988c4675-79q4p_namespace(556dc3d2-9da4-11ea-bca3-00163e01eb9a)" failed - error: 
command '/bin/kill -s SIGQUIT 1' exited with 1: kill: can't kill pid 1: Operation not permitted

Upvotes: 5

Janos Lenart
Janos Lenart

Reputation: 27100

From https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#debugging-hook-handlers

The logs for a Hook handler are not exposed in Pod events. If a handler fails for some reason, it broadcasts an event. [...] For PreStop, this is the FailedPreStopHook event. You can see these events by running kubectl describe pod <pod_name>. Here is some example output of events from running this command [...]

Upvotes: 0

Related Questions