CODEWITHSUNDEEP
pythonairflowdirected-acyclic-graphs
Sasha Podoprigora
Sasha Podoprigora

Reputation: 53

Force Airflow to terminate with exit code 1

Some of the tasks I run in Airflow occasionally fail for various reasons. Some I catch and others are new and aren't excepted. I want to get exit code 1 in order to receive a notification of such failures.

Below is an example of an error I didn't except, is there a way to have ValueError yield exit code 1? 

[2019-06-13 12:56:13,630] {bash_operator.py:127} INFO - ValueError: year 43631 is out of range
[2019-06-13 12:56:13,870] {bash_operator.py:127} INFO - 2019-06-13 12:56:13,869 - ODL - INFO - Closing db connection
[2019-06-13 12:56:13,870] {bash_operator.py:127} INFO - 2019-06-13 12:56:13,870 - ODL - INFO - End
[2019-06-13 12:56:13,990] {bash_operator.py:131} INFO - Command exited with return code 0
[2019-06-13 12:56:17,556] {logging_mixin.py:95} INFO - [2019-06-13 12:56:17,555] {jobs.py:2562} INFO - Task exited with return code 0

I also have other bits of code where the try block uses sys.exit(1). However, it is not raised, do I need to raise this error in order to exit with 1?

I've tried using sys.exit(1) in my try blocks. 

        if count==0:
            self._logger.error("Missing data, nothing was loaded")
            sys.exit(1)
        else:
            self._logger.info("Data is present, safe to load to prod!")

I expected to get the following in order to be notified of the failure. 

2019-06-13 12:56:17,556] {logging_mixin.py:95} INFO - [2019-06-13 12:56:17,555] {jobs.py:2562} INFO - Task exited with return code 1

Upvotes: 5

Views: 4704

Answers (1)

Daniel Huang
Daniel Huang

Reputation: 6548

It looks like you're using a BashOperator to call your Python script. This means you'll want to use something like set -e to ensure Bash stops execution when it hits the non-zero code on your Python command and it exits itself with the same code, see What does set -e mean in a bash script? for more details. Then this should fail the task as you'd expect.

Upvotes: 2

Related Questions