CODEWITHSUNDEEP
databricksdatabricks-workflows
Stanislav Žoldak
Stanislav Žoldak

Reputation: 115

Databricks Jobs - Is there a way to restart a failed task instead of the whole pipeline?

If I have for example a (multitask) Databricks job with 3 tasks in series and the second one fails - is there a way to start from the second task instead of running the whole pipeline again?

Upvotes: 5

Views: 5529

Answers (3)

cowbert
cowbert

Reputation: 3463

As of this writing (2024), using the Jobs 2.1 API to configure a job allows you to specify the following task-level attributes:

Note that condition_tasks do not support retries.

Upvotes: 1

Alex Ott
Alex Ott

Reputation: 87259

Right now this is not possible, but if you refer to the Databrick's Q3 (2021) public roadmap, there were some items around improving multi-task jobs.

Update: September 2022. This functionality was released back in May 2022nd with name repair & rerun

Upvotes: 5

Hubert Dudek
Hubert Dudek

Reputation: 1730

If you are running Databricks on Azure it is possible via Azure Data Factory.

Upvotes: 0

Related Questions