Airflow DAG failing - ERROR - HTTP Error 404: Not Found

Question

I'm a noob on Apache Airflow. Literally just started and I'm running into an error. I wrote my first dag, and am calling a Python script. It successfully ran and worked initially when I set it up, and I scheduled it to run once a day. I came to check it today and the dag failed with the message ERROR - HTTP Error 404: Not Found.

Everything is kinda new to me so apologies if this is an easy fix, but I don't understand why I'm getting a 404 Error. I've tried restarting docker to see if it was a webserver issue but no luck.

Thanks for the help

Screenshot

DAG

from airflow.models import DAG
from datetime import datetime, timedelta
from airflow.operators.python_operator import PythonOperator
from covid_cases import covid_data


default_args = {
    'owner': 'airflow',
    'start_date': datetime(2020, 10, 4),
    'retries': 2,
    'retry_delay': timedelta(seconds=20)}

dag =  DAG(dag_id = 'covid_updates',
           default_args = default_args,
           schedule_interval = "0 4 * * *")

t1 = PythonOperator(task_id = 'covid_update',
                    python_callable = covid_data,
                    dag = dag)

t1

PythonOperator - covid_cases.py

def covid_data():
    """

    Overview:
    ---------
    Downloads the USA COVID data directly from John Hopkins CSSEGISandData.
    This function merges all data from most current date to earliest date (2020-4-11).
    Using this function a user can conduct time series analysis in how COVID
    increases/decreases in various states.

    Output:
    -------
    One uncleaned .csv file called "usa_covid_cases.csv"


    """
    from datetime import datetime, timedelta
    import pandas as pd
    from urllib.error import HTTPError

    # Set starting index
    i = 1

    # Earliest dataset available on GitHub
    start_date = datetime.strptime('2020-4-11', '%Y-%m-%d').date()

    # Pulling today's date minus 1 day due to delay posting on GitHub
    today = datetime.now().date() - timedelta(days=i)

    # Setting llist to store dataframe file names
    file_names = []

    # Looping until date is equal to earlist date = Start Date
    while not (start_date.day == today.day and start_date.month == today.month and start_date.year == today.year):

        # Extracting variables from current date
        day = today.day
        month = today.month
        year = today.year

        # Cleaning and converting values for formatting on GitHub URL link
        if day < 10:
            day = '0' + str(day)

        if month < 10:
            month = '0' + str(month)

        # Setting variable for each url
        url = 'https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_daily_reports_us/{}-{}-{}.csv'\
                    .format(month, day, str(year))

        try:
            # Reading each url as a datafra,
            df = pd.read_csv(url, error_bad_lines=False)
        except HTTPError as e:
            # handle the error (print, log, etc)
            continue
        finally:
            # Code moved here to prevent an endless loop
            # Subtracting the new index to increase 1 less day from the current date
            today = datetime.now().date() - timedelta(days=i)

        # Saving each dataframe into the empty list
        file_names.append(df)

        # Increasing the index by 1
        i += 1

    # Once while loop ends - concat all the files into a single dataframe
    new_df = pd.concat(file_names)

    # Save output into new csv file
    new_df.to_csv('usa_covid_cases.csv')

Answer 1

I just took a look at the GitHub repo and the earliest dataset is from 04-12-2020.

To prevent failing DAGs from possible deleted datasets you can wrap your pd.read_csv() in a try except block, like this:

# Import the HTTPError (error from you screenshot)
from urllib.error import HTTPError
...

try:
    # Reading each url as a datafra,
    df = pd.read_csv(url, error_bad_lines=False)
except HTTPError as e:
    # handle the error (print, log, etc)
    continue
finally:
    # Code moved here to prevent an endless loop
    # Subtracting the new index to increase 1 less day from the current date
    today = datetime.now().date() - timedelta(days=i)

 # Saving each dataframe into the empty list
 file_names.append(df)

 # Increasing the index by 1
 i += 1

...

Answer 2

From the logs, the error happens when you are trying to read data from a URL and that URL does not exists.

pd.read_csv(url, error_bad_lines=False) #Line 50 of covid_data.py