Apache Airflow - Was Working Fine Now Says Log File Isn't Local Error & Exceptions are Popping Up

Question

So it looks like my install of apache airflow on a Google Compute Engine instance broke down. Everything was working great and then two days ago all the DAG runs show up stuck in a running state. I am using LocalExecutioner.

When I try to look at the log I get this error:

* Log file isn't local. * Fetching here: http://:8793/log/collector/aa_main_combined_collector/2017-12-15T09:00:00 *** Failed to fetch log file from worker.

I didn't touch a setting anywhere. I looked through all the config files and I scanned the logs and I see this error

[2017-12-16 20:08:42,558] {jobs.py:355} DagFileProcessor0 ERROR - Got an exception! Propagating... Traceback (most recent call last): File "/usr/local/lib/python3.4/dist-packages/airflow/jobs.py", line 347, in helper pickle_dags) File "/usr/local/lib/python3.4/dist-packages/airflow/utils/db.py", line 53, in wrapper result = func(*args, **kwargs) File "/usr/local/lib/python3.4/dist-packages/airflow/jobs.py", line 1584, in process_file self._process_dags(dagbag, dags, ti_keys_to_schedule) File "/usr/local/lib/python3.4/dist-packages/airflow/jobs.py", line 1173, in _process_dags dag_run = self.create_dag_run(dag) File "/usr/local/lib/python3.4/dist-packages/airflow/utils/db.py", line 53, in wrapper result = func(*args, **kwargs) File "/usr/local/lib/python3.4/dist-packages/airflow/jobs.py", line 763, in create_dag_run last_scheduled_run = qry.scalar() File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/query.py", line 2843, in scalar ret = self.one() File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/query.py", line 2814, in one ret = self.one_or_none() File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/query.py", line 2784, in one_or_none ret = list(self) File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/query.py", line 2855, in iter return self._execute_and_instances(context) File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/query.py", line 2878, in _execute_and_instances result = conn.execute(querycontext.statement, self._params) File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/engine/base.py", line 945, in execute return meth(self, multiparams, params) File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement compiled_sql, distilled_params File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context context) File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/engine/base.py", line 1405, in _handle_dbapi_exception util.reraise(*exc_info) File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/util/compat.py", line 187, in reraise raise value File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context context) File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/engine/default.py", line 470, in do_execute cursor.execute(statement, parameters) File "/usr/local/lib/python3.4/dist-packages/airflow/bin/cli.py", line 69, in sigint_handler sys.exit(0) SystemExit: 0

Any thoughts out there?

Answer 1

Look at the log URL, verify if it ends with a date with special characters +:

&execution_date=2018-02-23T08:00:00+00:00

This was fixed here.

You can replace the + for -, or replace all special characters, in my case:

&execution_date=2018-02-23T08%3A00%3A00%2B00%3A00

This happens here.

The FileTaskHandler can not load the log from local disk, and try to load from worker.

Another thing that could be causing this error is the exclusion of the airflow/logs folder or the sub folders inside it.

Answer 2

I solved this problem though in doing so I discovered another problem.

Long and short of it as soon as I manually started the scheduler, everything worked again. It appears the problem was that the scheduler did not get restarted correctly after a system reboot.

I have scheduler running through SystemD. The Webserver .service works fine. However I do notice that the scheduler .service continually restarts. It appears there is an issue there I need to resolve. This part of it is solved for now.