CODEWITHSUNDEEP
airflow
Joyce
Joyce

Reputation: 1451

How can one set a variable for use only during a certain dag_run

How do I set a variable for use during a particular dag_run. I'm aware of setting values in xcom, but not all the operators that I use has xcom support. I also would not like to store the value into the Variables datastore, in case another dag run begins while the current one is running, that need to store different values.

Upvotes: 1

Views: 1655

Answers (1)

y2k-shubham
y2k-shubham

Reputation: 11597

The question is not clear, but from whatever I can infer, I'll try to clear your doubts

not all the operators that I use has xcom support

Apparently you've mistaken xcom with some other construct because xcom feature is part of TaskInstance and the functions xcom_push() and xcom_pull() are defined in BaseOperator itself (which is the parent of all Airflow operators)

I also would not like to store the value into the Variables datastore, in case another dag run begins while the current one is running, that need to store different values.

It is straightforward (and no-brainer) to separate-out Variables on per-DAG basis (see point (6)); but yes for different DagRuns of a single DAG, this kind of isolation would be a challenge. I can think of xcom to be the easiest workaround for this. Have a look at this for some insights on usage of Xcoms.

Additionally, if you want to manipulate Variables (or any other Airflow model) at runtime (though I would recommend you to avoid it particularly for Variables), Airflow also gives completely liberty to exploit the underlying SQLAlchemy ORM framework for that. You can take inspiration from this.

Upvotes: 1

Related Questions