CODEWITHSUNDEEP
pythonpython-2.7luigi
anumehta
anumehta

Reputation: 11

How to make a Luigi task shut itself down?

We have a Luigi task that starts at 2am and can run for an arbitrary number of hours. We want to make sure that if it isn't done by, say 6am, it shuts itself down. Is there any way to do this? The task runs on Mesos, so I know that I can run a separate task or script to use the Mesos executor HTTP API and kill it, but is there a way to code the task itself to shutdown at a given time?

Thanks in advance.

Upvotes: 1

Views: 1741

Answers (1)

matagus
matagus

Reputation: 6206

Use a custom timeout for your task. According to the docs you can set it at luigi level o per task:

[worker]

timeout:

Number of seconds after which to kill a task which has been running for too long. This provides a default value for all tasks, which can be overridden by setting the worker-timeout property in any task. This only works when using multiple workers, as the timeout is implemented by killing worker subprocesses. Default value is 0, meaning no timeout.

So, your task should look like this:

class MyTask(luigi.Task):
    worker_timeout = 60 * 60 * 4 # 4 hours in seconds

    # (rest of your task)
    # def run():
    #    do_something()

Upvotes: 2

Related Questions