CODEWITHSUNDEEP
pythonceleryairflow
Butterfly Coder
Butterfly Coder

Reputation: 307

Airflow error during celery execution

We have an airflow setup with rabbitmq, mysql db and celery executor setup in a docker instance. Trigerring the first individual step/opertator via a run command as below works.

airflow run wfm_model_training_client_1_queue_11 fetch_top_funnel_data  2016-03-03

However when we trigger the entire dag via a backfill command as below, we encounter the following error during celery execution. Clearly this is an error with celery, however how do i extract more logs to debug the same.

backfill command

airflow backfill wfm_model_training_client_1_queue_11 -s 2016-03-03 -e 2016-03-03

Celery error in celery_taskmeta table in traceback column

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
    return self.run(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/airflow/executors/celery_executor.py", line 59, in execute_command
    except subprocess.CalledProcessError as e:
AirflowException: Celery command failed

Upvotes: 1

Views: 3065

Answers (1)

Butterfly Coder
Butterfly Coder

Reputation: 307

The error was due to some pickle related issue. I bypassed this issue by passing the --donot_pickle args to the backfill command.

It was resolved via the updated command

airflow run wfm_model_training_client_1_queue_11 fetch_top_funnel_data  2016-03-03 --donot_pickle

Upvotes: 2

Related Questions