Reputation: 23
I'd like to get information about k8s cronjob time. There are so many jobs in my k8s program. So It's hard to count what time there are focused on. I want to distribute my jobs evenly. Is there are way to count cronjob time or sort by time?
Upvotes: 0
Views: 6320
Reputation: 4614
I have tried to find a suitable tool that can help with your case. Unfortunately, I did not find anything suitable and easy to use at the same time.
It is possible to use Prometheus + Grafana
to monitor CronJobs
e.g using this Kubernetes Cron and Batch Job monitoring dashboard.
However, I don't think you will find any useful information in this way, just a dashboard that displays the number of CronJobs
in the cluster.
For this reason, I decided to write a Bash script that is able to display the last few CronJobs
run in a readable manner.
As described in the Kubernetes CronJob documentation:
A CronJob creates Jobs on a repeating schedule.
To find out how long a specific Job was running, we can check its startTime
and completionTime
e.g. using the commands below:
# kubectl get job <JOB_NAME> --template '{{.status.startTime}}' # "startTime"
# kubectl get job <JOB_NAME> --template '{{.status.completionTime}}' # "completionTime"
To get the duration of Jobs
in seconds, we can convert startTime
and completionTime
dates to epoch:
# date -d "<SOME_DATE> +%s
And this is the entire Bash script:
NOTE: We need to pass the namespace name as an argument.
#!/bin/bash
# script name: cronjobs_timetable.sh <NAMESPACE>
namespace=$1
for cronjob_name in $(kubectl get cronjobs -n $namespace --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}'); do
echo "===== CRONJOB_NAME: ${cronjob_name} ==========="
printf "%-15s %-15s %-15s %-15s\n" "START_TIME" "COMPLETION_TIME" "DURATION" "JOB_NAME"
for job_name in $(kubectl get jobs -n $namespace --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' | grep -w "${cronjob_name}-[0-9]*$"); do
startTime="$(kubectl get job ${job_name} -n $namespace --template '{{.status.startTime}}')"
completionTime="$(kubectl get job ${job_name} -n $namespace --template '{{.status.completionTime}}')"
if [[ "$completionTime" == "<no value>" ]]; then
continue
fi
duration=$[ $(date -d "$completionTime" +%s) - $(date -d "$startTime" +%s) ]
printf "%-15s %-15s %-15s %-15s\n" "$(date -d ${startTime} +%X)" "$(date -d ${completionTime} +%X)" "${duration} s" "$job_name"
done
done
By default, this script only displays the last three Jobs
, but it may by modified in the Job configuration using the .spec.successfulJobsHistoryLimit
and .spec.failedJobsHistoryLimit
fields (for more information see Kubernetes Jobs History Limits)
We can check how it works:
$ ./cronjobs_timetable.sh default
===== CRONJOB_NAME: hello ===========
START_TIME COMPLETION_TIME DURATION JOB_NAME
02:23:00 PM 02:23:12 PM 12 s hello-1616077380
02:24:02 PM 02:24:13 PM 11 s hello-1616077440
02:25:03 PM 02:25:15 PM 12 s hello-1616077500
===== CRONJOB_NAME: hello-2 ===========
START_TIME COMPLETION_TIME DURATION JOB_NAME
02:23:01 PM 02:23:23 PM 22 s hello-2-1616077380
02:24:02 PM 02:24:24 PM 22 s hello-2-1616077440
02:25:03 PM 02:25:25 PM 22 s hello-2-1616077500
===== CRONJOB_NAME: hello-3 ===========
START_TIME COMPLETION_TIME DURATION JOB_NAME
02:23:01 PM 02:23:32 PM 31 s hello-3-1616077380
02:24:02 PM 02:24:34 PM 32 s hello-3-1616077440
02:25:03 PM 02:25:35 PM 32 s hello-3-1616077500
===== CRONJOB_NAME: hello-4 ===========
START_TIME COMPLETION_TIME DURATION JOB_NAME
02:23:01 PM 02:23:44 PM 43 s hello-4-1616077380
02:24:02 PM 02:24:44 PM 42 s hello-4-1616077440
02:25:03 PM 02:25:45 PM 42 s hello-4-1616077500
Additionally, you'll likely want to create exceptions and error handling to make this script work as expected in all cases.
Upvotes: 3