CODEWITHSUNDEEP
pythonpython-multiprocessingorchestrationluigi
sweeeeeet
sweeeeeet

Reputation: 1819

parallelizing tasks in Luigi Orchestrator

I defined three tasks T1, T2, and T3, and then a task T4 as follows:

class T4(luigi.Task)
    def requires(self):
        return [T1(), T2(), T3()]

Is there a natural way to tell Luigi that I want these tasks T1, T2, and T3 to be executed in parallel?

Upvotes: 11

Views: 6436

Answers (1)

matagus
matagus

Reputation: 6206

It depends on what dependencies T1, T2 and T3 have. If they haven't another task as a common dependency, you can just run your task specifying --workers=3 and Luigi will run each task in a separate worker.

Upvotes: 18

Related Questions