CODEWITHSUNDEEP
airflowdirected-acyclic-graphs
Shubhank Gupta
Shubhank Gupta

Reputation: 825

View on_failure_callback DAG logger

Let's take an example DAG.

DAG Graph

Here is the code for it.

import logging
from airflow import DAG
from datetime import datetime, timedelta
from airflow.models import TaskInstance
from airflow.operators.python import PythonOperator
from airflow.operators.dummy import DummyOperator


def task_failure_notification_alert(context):
    logging.info("Task context details: %s", str(context))


def dag_failure_notification_alert(context):
    logging.info("DAG context details: %s", str(context))


def red_exception_task(ti: TaskInstance, **kwargs):
    raise Exception('red')


default_args = {
    "owner": "analytics",
    "start_date": datetime(2021, 12, 12),
    'retries': 0,
    'retry_delay': timedelta(),
    "schedule_interval": "@daily"
}

dag = DAG('logger_dag',
          default_args=default_args,
          catchup=False,
          on_failure_callback=dag_failure_notification_alert
          )

start_task = DummyOperator(task_id="start_task", dag=dag, on_failure_callback=task_failure_notification_alert)
red_task = PythonOperator(
    dag=dag,
    task_id='red_task',
    python_callable=red_exception_task,
    provide_context=True,
    on_failure_callback=task_failure_notification_alert
)
end_task = DummyOperator(task_id="end_task", dag=dag, on_failure_callback=task_failure_notification_alert)

start_task >> red_task >> end_task

We can see two functions i.e. task_failure_notification_alert and dag_failure_notification_alert are being called in case of failures. We can see logs in case of Task failure by the below steps.

We can see logs for the task as below.

Task options screen

Logs screen

but I am unable to find logs for the on_failure_callback of DAG anywhere in UI. Where can we see it?

Upvotes: 3

Views: 3502

Answers (1)

ozs
ozs

Reputation: 3681

Under airflow/logs find the "scheduler" folder, under it look for the specific date you ran the Dag for example 2022-12-03 and there you will see name of the dag_file.log.

Upvotes: 3

Related Questions