CODEWITHSUNDEEP
pythondatabricks
desap
desap

Reputation: 205

Databricks job not failing after Exception

I have a Databricks job which runs my Python notebook. I am running Databricks 7.4 lts. This job is triggered via Powershell.

If something goes wrong during the pipeline I raise an Exception. This exception is not caught. However, the job does not show as "Failed" after the exception is raised. Instead, the job shows "Successful". I can see the exception in the logs for the job.

Why is the job not showing as failed?

if pipeline_failure:
    raise Exception("Job should show failed")

Upvotes: 2

Views: 1931

Answers (1)

desap
desap

Reputation: 205

In the Databricks Python notebook I was calling display(dataframe) and raise Exception() in the same cell.

Specifically, display() was right before Exception()

After moving display(dataframe) out of the same cell as the Exception() statement the Databricks job to show "Failed" when the exception was raised.

Upvotes: 1

Related Questions