CODEWITHSUNDEEP
flyte
Michael Sun
Michael Sun

Reputation: 23

Prevent one failed subtask failing all tasks in Flyte

I have a dynamic_task which kicks off a number of python_tasks. However, as soon as one of the python_tasks fails, the other ones that are still running would fail as well. Is this by design? Is there a way to change this behavior so that other tasks can still complete without failing?

Upvotes: 1

Views: 401

Answers (2)

Ketan Umare
Ketan Umare

Reputation: 46

The idea is that partial failures are not tolerated within a data passing DAG. If some node fails, then by definition the data is partial.

But for dynamic array tasks, Flyte allows a special provision (actually the Array tasks plugin), which allows the users to provide a ratio of acceptable successful tasks.

Upvotes: 0

kaylindris
kaylindris

Reputation: 88

This is by design, as a means to save resources, but it is configurable. Presumably, dynamic tasks are related to each other, and downstream tasks will need the output of all of them. So if one fails, the default behavior is to fail the rest.

If you'd like to change this, create your dynamic task with a float as this argument in the decorator: https://github.com/lyft/flytekit/blob/d4cfedc4c580f08bf904e6e474a0b948a4608737/flytekit/common/tasks/sdk_dynamic.py#L84

Upvotes: 1

Related Questions