CODEWITHSUNDEEP
airflow
partlov
partlov

Reputation: 14277

Changes in DAG/Operator while DAG is running

  1. What would happen if DAG changes while DAG is running (particularly if this is dynamic DAG)
  2. What would happen if code of custom made operator is changed during some DAG run

Upvotes: 7

Views: 2014

Answers (1)

partlov
partlov

Reputation: 14277

So after some investigation I came to conclusion that DAG changes will be visible in current DAG run and that operators (and all other plugins) are not reloaded by default.

  1. DAG changes during DAG run

    • If you remove task while DAG is running scheduler notice that, set status for that task to "Removed" while logging current line:

    State of this instance has been externally set to removed. Taking the poison pill.

    • If you add new task it will also be visible and executed. It will be executed even if it is upstream task of task which is already finished

    • If task is changed, changes will be incorporated into current DAG run only if task is not already started execution or finished

  2. All plugins (included operators) are not refreshed automatically by default. You can restart Airflow or you can set reload_on_plugin_change property in [webserver] section of airflow.cfg file to True

Upvotes: 6

Related Questions