CODEWITHSUNDEEP
airflow
Desiree Cox
Desiree Cox

Reputation: 359

Possible to set different executor for each Airflow DAG?

I am looking to add another DAG to an existing Airflow server. The server is currently using LocalExecutor but I might want my DAG to use CeleryExecutor. It seems like the configuration file airflow.cfg only allows one executor:

# The executor class that airflow should use. Choices include
# SequentialExecutor, LocalExecutor, CeleryExecutor
executor = LocalExecutor

Is it possible to configure Airflow such that the existing DAGs can continue to use LocalExecutor and my new DAG can use CeleryExecutor or a custom executor class? I haven't found any examples of people doing this nor come across anything in the Airflow documentation.

Upvotes: 8

Views: 3269

Answers (2)

mithos
mithos

Reputation: 61

If you have a SubDAG within your DAG, you can pass in a specific executor to that SubDagOperator. For instance, to use a SequentialExecutor:

bar_subdag = SubDagOperator(
    task_id='bar',
    subdag=my_subdag('foo', 'bar', default_args),
    default_args=default_args,
    dag=foo_dag,
    executor=SequentialExecutor()
)

This is on 1.8, not sure if 1.9 is different.

Upvotes: 6

Scott Ding
Scott Ding

Reputation: 574

Seems the scheduler will only start one instance of the executor.

Upvotes: 1

Related Questions