CODEWITHSUNDEEP
python-3.xairflow
KristiLuna
KristiLuna

Reputation: 1903

SubDagOperator Invalid arguments were: **kwargs: {'executor': LocalExecutor(parallelism=32)}

We're upgrading to Airflow 2.0 and I have the below task:

with dag:
    cms_ingest = SubDagOperator(
        subdag=cms_s3ingest(
            DAG_NAME, 'cms_s3ingest', default_args['start_date'], dag.schedule_interval),
        task_id='cms_s3ingest',
        # so that subtasks can run in parallel
        executor=LocalExecutor(),
        task_concurrency=4)

but I'm seeing this error:

airflow.exceptions.AirflowException: Invalid arguments were passed to SubDagOperator (task_id: cms_s3ingest). Invalid arguments were: **kwargs: {'executor': LocalExecutor(parallelism=32)}

in my airflow.cfg file I have the value: parallelism = 32

which I believe is what LocalExecutor is using https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/executors/local_executor/index.html#airflow.executors.local_executor.LocalExecutor.

Why is this error populating in Airflow 2.0.2, and how should I fix it?

Upvotes: 1

Views: 3321

Answers (1)

Elad Kalif
Elad Kalif

Reputation: 15979

The executor parameter was removed from SubDagOperator when the behavior was changed from backfil to the scheduler logic (PR).

But even before that it wasn't recommended to parallel tasks inside SubDagOperator which is why it was defaulted to SequentialExecutor (you can read about the reasons here)

Since you are using Airflow>2.0.0 note that SubDags are deprecated. You should move to TasksGroups.

Upvotes: 1

Related Questions