CODEWITHSUNDEEP
airflow
lampShadesDrifter
lampShadesDrifter

Reputation: 4159

Would airflow tasks for different dags have race condition if changing env vars with same name?

What would happen if had airflow dags running concurrently where tasks (say BashOperators) each modified the same env variable? Would both tasks see the change from the other? Would airflow run into race conditions or would the env changes only affect the scripts in isolation?

I would normally think that the env changes would be self contained in this case, but asking because did have a problem with ariflow in the past where setting os.environ[...] values (at the dag definition level) affected all dags.

* Could not currently test myself because having other problems setting up LocalExecutor mode.

Upvotes: 1

Views: 481

Answers (1)

lampShadesDrifter
lampShadesDrifter

Reputation: 4159

From discussions on the airflow email list, learned that all of the Airflow executors use subprocesses (at minimum) for launching tasks. Thus, setting environment variables will not propagate back "up" to the parent process.

This can also be found in the docs:

The scheduler starts an instance of the executor specified in the your airflow.cfg. If it happens to be the airflow.contrib.executors.local_executor.LocalExecutor, tasks will be executed as subprocesses; in the case of airflow.executors.celery_executor.CeleryExecutor, airflow.executors.dask_executor.DaskExecutor, and airflow.contrib.executors.mesos_executor.MesosExecutor, tasks are executed remotely.

Upvotes: 0

Related Questions