CODEWITHSUNDEEP
pythonairflowdirected-acyclic-graphsraiseairflow-taskflow
S_2103
S_2103

Reputation: 1

How to make a DAG fail when we are raising exception in our python code?

I am new to Airflow DAGs. I have a python script to fetch API calls. I have added a raise Exception in my code, if all API calls failed i.e. status_code!=200, then it should raise exception. However, in airflow logs, it is shown as INFO - [base] An error occurred: All API calls failed for all IDs and DAG succeeds. How do I make the DAG failed if all API calls fail.

Below is the sample code I have:

        response = requests.get(url)
        data = response.json()
        if response.status_code!=200:
            print(f"Error occurred for ID - Response code {response.status_code} {response.reason}")
            break
        if response.status_code==200:
            df1 = pd.concat([df1, pd.json_normalize(data)])
        else:
            print(f'No data to fetch data for {reaction}:{id}')


if not df1.empty:  
    all_accounts_failed = False

if all_accounts_failed:
    raise Exception("All API calls failed for all IDs")
    raise
else:
    try:
      #rest of the code

Expecting the DAG to be failed however DAG succeeded with just showing it as INFO

Snippet from DAG

Upvotes: 0

Views: 36

Answers (1)

bcincy
bcincy

Reputation: 31

There are two paths to solving this:

  1. You can update your code to raise AirflowException. Here is a link to Airflow Exceptions. You'll need to add to the imports from airflow.exceptions import AirflowException to use it. Raising an Airflow Exception will cause the DAG to fail.

  2. You can convert your requests to using the HttpHook, which will throw an AirflowException if the response is not a 200 response. Here is a link to the hook.

Upvotes: 0

Related Questions