CODEWITHSUNDEEP
airflow
bronzels
bronzels

Reputation: 1393

Airflow DAG "seems to be existing only locally. The master scheduler doesn't seem to be aware of its existence"

I used airflow for workflow of Spark jobs. After installation, I copy the DAG files into DAGs folder set in airflow.cfg. I can backfill the DAG to run the BashOperators successfully. But there is always a warning like the one mentioned. I didn't verify if the scheduling is fine, but I doubt scheduling can work as the warning said the master scheduler doesn't know of my DAG's existence. How can I eliminate this warning and get scheduling work? Anybody run into the same issue who can help me out?

Upvotes: 13

Views: 10217

Answers (1)

tobi6
tobi6

Reputation: 8239

This is usually connected to the Scheduler not running or the refresh interval being too wide. There are no log entries present so we cannot analyze from there. Also, unfortunately the very cause might have been ignored, because this is usually the root of the problem:

I didn't verify if the scheduling is fine.

So first you should check if both of the following services are running:

airflow webserver

and

airflow scheduler

If that won't help, see this post for more reference: Airflow 1.9.0 is queuing but not launching tasks

Upvotes: 22

Related Questions