CODEWITHSUNDEEP
pythonpython-3.xairflow
aaron
aaron

Reputation: 6489

Still unable to run all tasks in parallel with Airflow

I have task that is set up as in this question.

Based on the UI it looks like the dependencies are well-defined:

enter image description here

I have tested individual tasks as follows: airflow test capone_dash_preproc AAAG5608078M2 2017-07-25. This works great, updating the target database entries corresponding to that user. When I try to run the full task, however, it seems like it's getting hung up. python3 dash_dag.py basically results in the following terminal output on the web server, repeated endlessly. All CPUs are quiet, so it doesn't seem like there's much calculation happening: 

/usr/local/lib/python3.6/site-packages/flask/exthook.py:71: ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
  .format(x=modname), ExtDeprecationWarning
[2017-07-25 16:48:23,266] [58627] {models.py:167} INFO - Filling up the DagBag from /Users/aaronpolhamus/airflow/dags
[2017-07-25 16:48:25 -0500] [58487] [INFO] Handling signal: ttou
[2017-07-25 16:49:13 -0500] [58557] [INFO] Worker exiting (pid: 58557)
[2017-07-25 16:49:44 -0500] [58487] [INFO] Handling signal: ttin
[2017-07-25 16:49:44 -0500] [58642] [INFO] Booting worker with pid: 58642
/usr/local/lib/python3.6/site-packages/flask/exthook.py:71: ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
  .format(x=modname), ExtDeprecationWarning
[2017-07-25 16:49:44,607] [58642] {models.py:167} INFO - Filling up the DagBag from /Users/aaronpolhamus/airflow/dags
[2017-07-25 16:49:46 -0500] [58487] [INFO] Handling signal: ttou
[2017-07-25 16:50:21 -0500] [58568] [INFO] Worker exiting (pid: 58568)
[2017-07-25 16:50:51 -0500] [58487] [INFO] Handling signal: ttin
[2017-07-25 16:50:51 -0500] [58661] [INFO] Booting worker with pid: 58661
/usr/local/lib/python3.6/site-packages/flask/exthook.py:71: ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cache instead.
  .format(x=modname), ExtDeprecationWarning
[2017-07-25 16:50:52,324] [58661] {models.py:167} INFO - Filling up the DagBag from /Users/aaronpolhamus/airflow/dags
[2017-07-25 16:50:54 -0500] [58487] [INFO] Handling signal: ttou
[2017-07-25 16:51:20 -0500] [58596] [INFO] Worker exiting (pid: 58596)
[2017-07-25 16:51:50 -0500] [58487] [INFO] Handling signal: ttin
[2017-07-25 16:51:50 -0500] [58677] [INFO] Booting worker with pid: 58677
...

My confusion arises from the fact that the individual tests run just fine and populate the database. It is the entire run that hangs and fails. Anything obvious here?

Upvotes: 0

Views: 1826

Answers (1)

Bolke de Bruin
Bolke de Bruin

Reputation: 760

Airflow exists of multiple components. The most important being the scheduler, webserver, and worker (in case of horizontal scaling). The scheduler always* needs to be running to have Airflow execute tasks. The webserver is only there to provide a nice UI on top of what the scheduler is doing and for the Rest API. All components use the backing database.

You are showing the output of the webserver, while I would have expected to see the output of the scheduler. From the screenshot you posted and the way you try running your DAG my assumption is that you did not start it.

So start the scheduler by issuing airflow scheduler on the command line and your DAG will probably kick off.

*backfills are an exception

Upvotes: 1

Related Questions