CODEWITHSUNDEEP
airflow
Aviv Goldgeier
Aviv Goldgeier

Reputation: 809

Yet another “This DAG isn't available in the webserver DagBag object ”

This seems to be a fairly common issue. I have a DAG where, not only can I trigger it manually with airflow trigger_dag, but it's even executing according to its schedule, but it refuses to show up in the UI.

I've already, restarted the webserver and scheduler multiple times, pressed "refresh" like a billion times, and ran it through airflow backfill. Anyone have any other ideas? Any other pertinent information I can provide?

I'm on Airflow 1.9.0.

Upvotes: 2

Views: 1061

Answers (1)

knutole
knutole

Reputation: 1787

I have been debugging this exact problem for the last few hours. It seems to be due to a silent error in the DAG. Leaving my notes here for the next poor soul.

So in my case, this error was due to the following blocks of code in my DAG:

This fails:

def read_lakes_id_file_simple():
    LAKES_ID_FILE = "/home/airflow/gcs/data/lakes_to_monitor.json"
    with open(LAKES_ID_FILE) as json_file:
        data = json.load(json_file)
    return data

This passes:

def read_lakes_id_file_simple():
    try:
        LAKES_ID_FILE = "/home/airflow/gcs/data/lakes_to_monitor.json"
        with open(LAKES_ID_FILE) as json_file:
            data = json.load(json_file)
        return data
    except Exception as e:
        return 'LOTS OF LAKES'

So I'm guessing the first fails somehow when read/checked by scheduler, perhaps, as it can't find the file, or whatnot, while the second succeeds because it's run in the right path by the worker. (Or it could be something else.) What seems clear is that there are two different runs and behaviors when loading/running the DAG, and one fails silently, while the other succeeds.

This leads to bizarre behavior, such a as the DAG running fine the first time, then disappearing from the Airflow Web Interface afterwards.

So my suggestion to you is to add try/except to anything that might fit the bill, as a way of debugging your code.

Upvotes: 1

Related Questions