CODEWITHSUNDEEP
airflow
Shengxin Huang
Shengxin Huang

Reputation: 707

How ro remove unwanted broken DAGs in airflow

I write something wrong in my sql_test.py,and run python sql_test.py,the error is 'no module named xxx',and in web-ui it shows a red error - Broken DAG. And then I run airflow list_dags the same error occurs again .This is strange and I don't know what's happening. I tried to run airflow delete_dags sql_test but there is no such id. How can I :

  1. remove the waning in web-ui
  2. get sql_test out of list_dags

Upvotes: 2

Views: 4021

Answers (2)

FraDel
FraDel

Reputation: 184

If you don't want to repair your DAG and ignore it, you can remove the unwanted DAG by specifying the DAG's underlying file in an .airflowignore file.

Upvotes: 1

y2k-shubham
y2k-shubham

Reputation: 11597

There's some syntactical mistake in your dag-definition file, resulting in failure in parsing the DAG. When Airflow fails to parse a DAG, several functionalities get broken (like list_dags in your case)

Of course deleting the problematic dag-definition file would fix it, but that's not a solution. So here's how you can understand what's wrong and fix it

  • From linux shell, go to Airflow's logs folder

    cd $AIRFLOW_HOME/logs/scheduler/latest/

  • Run tree command to see directory structure

    tree -I "__init__.py|__pycache__|*.pyc"

  • View the last few lines of the log file of your corresponding broken dag

    tail -n 25 /path/to/my/broken-dag.py.log

This will give you the stack-trace that Airflow threw while trying to parse your broken dag file. That would hopefully help you diagnose the problem and patch it.

Once your dag-definition file is fixed

  • the broken dag message would disappear from UI
  • DAG would appear in the UI (refresh it a few times)
  • list_dags command would also start working

Upvotes: 2

Related Questions