CODEWITHSUNDEEP
kubernetes
Võ Trường Duy
Võ Trường Duy

Reputation: 121

How to know if Kubernetes job is done?

I use Python kubernetes-client, and want to wait if the job is done:

api_instance.create_namespaced_job("default", body, pretty=True)

This call just makes a submit job, it will return the response even though the job is still running. How can I wait for the job to finish?

Upvotes: 5

Views: 4969

Answers (4)

patm
patm

Reputation: 136

Hopefully this is helpful to someone. A function that gets pods based on status, job name, or both, then to answer the original question, wait for length of get_pods(job_name=job_name, status='running') to == 0.

from kubernetes import client
from time import sleep

configuration = client.Configuration()
configuration.host = "{your_url}"
configuration.api_key['authorization'] = f"Bearer {your_token}"
configuration.verify_ssl = False
apiclient = client.ApiClient(configuration)
namespace = "{your_namespace}"
kube = client.CoreV1Api(apiclient)
        

def get_pods(job_name=None, status=None):

    if job_name and status:
        return kube.list_namespaced_pod(namespace, field_selector=f"status.phase={status.capitalize()}", label_selector=f"job-name={job_name}").items

    if job_name:
        return kube.list_namespaced_pod(namespace, label_selector=f"job-name={job_name}").items

    if status:
        return kube.list_namespaced_pod(namespace, field_selector=f"status.phase={status.capitalize()}").items

    return kube.list_namespaced_pod(namespace).items


while len(get_pods(job_name="{your_job_name}", status='running')) > 0:
    sleep(10)

Upvotes: 0

Utkarsh Shende
Utkarsh Shende

Reputation: 1

from kubernetes import client, watch

def wait_for_job(self):
    logger.info("Waiting for job to complete...")
    w = watch.Watch()
    # timeout_seconds=0 -> will run watch for infinite time, but doesn't raise exception when wait is over
    # _request_timeout=timeout -> it is max timeout for request
    for event in w.stream(
        client.BatchV1Api().list_namespaced_job,
        namespace=self.namespace,
        label_selector=f"job-name={self.job_name}",
        timeout_seconds=0,
        _request_timeout=self.active_deadline_seconds
    ):
        o = event["object"]

        print(o.metadata.name)
        if o.status.succeeded:
            w.stop()
            return

        if not o.status.active and o.status.failed:
            w.stop()
            raise Exception("Job Failed")

Upvotes: 0

smrt28
smrt28

Reputation: 469

I found the solution. You can recognize the job is complete by watching the jobs and observing the events:

from kubernetes import client, config, watch

config.load_kube_config()
api_client = client.BatchV1Api()
print("INFO:    Waiting for event to come up...")
w = watch.Watch()
for event in w.stream(api_client.list_job_for_all_namespaces):
    o = event['object']
    print(o)
    if (o.status.... = "Complete"): ....

Upvotes: 3

smrt28
smrt28

Reputation: 469

The only way I managed to do this is calling in the loop:

api_instance.read_namespaced_job_status()

...and check the result status.

You can reach the same by calling kubectl wait but it's not python. You can clone kubectl sources (in Go) and find how the hell they do it there.

Upvotes: 0

Related Questions