CODEWITHSUNDEEP
celeryairflowairflow-scheduler
Rafiul Sabbir
Rafiul Sabbir

Reputation: 636

Apache Airflow celery executor is not getting result backend

I am running Apache Airflow version 1.9.0 and when I try to run a task from UI, I get the following error in airflow scheduler console:

[2018-05-08 12:09:06,737] {jobs.py:1077} INFO - No tasks to consider for execution.
[2018-05-08 12:09:06,738] {jobs.py:1662} INFO - Heartbeating the executor
[2018-05-08 12:09:06,738] {celery_executor.py:101} ERROR - Error syncing the celery executor, ignoring it:
[2018-05-08 12:09:06,738] {celery_executor.py:102} ERROR - No result backend configured.  Please see the documentation for more information.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/airflow/executors/celery_executor.py", line 83, in sync
    state = async.state
  File "/usr/local/lib/python2.7/dist-packages/celery/result.py", line 329, in state
    return self.backend.get_status(self.id)
  File "/usr/local/lib/python2.7/dist-packages/celery/backends/base.py", line 547, in _is_disabled
    'No result backend configured.  '
NotImplementedError: No result backend configured.  Please see the documentation for more information.

In my airflow.cfg, I have the following variables in [celery] section:

celery_app_name = airflow.executors.celery_executor
celeryd_concurrency = 16
worker_log_server_port = 8795
broker_url = amqp://guest:guest@localhost:5672//
celery_result_backend = amqp://guest:guest@localhost:5672//
flower_host = 0.0.0.0
flower_port = 5555
default_queue = default

What am I doing wrong here?

Upvotes: 0

Views: 5826

Answers (1)

Hito
Hito

Reputation: 840

You should not point celery_result_backend to a RabbitMQ instance since the purpose of this backend is to store information concerning the status of the tasks and RabbitMQ is not the right tool for that (Please correct me if I'm mistaken).

You can use Redis in case you want to keep using the same instance as broker and backend, or alternatively you can use postgres as the backend which I recommend. A sample configuration for Postgres would be the following:

celery_result_backend = db+postgresql://airflow:****@postgres/airflow

More info on the official docs: Here

Upvotes: 2

Related Questions