CODEWITHSUNDEEP
airflowairflow-2.x
elaspog
elaspog

Reputation: 1709

Is it possible to get multiple values from XCOM for a single task at once in Airflow?

Is it possible to retrieve multiple values from XCOM, pushed by a single task but with different keys?

I think I've seen examples to this:

# pulls one value
pulled_value = ti.xcom_pull(key=None, task_ids='my_task_id')

and to this:

# pulls multiple values but from multiple tasks
pulled_value_1, pulled_value_2 = ti.xcom_pull(key=None, task_ids=['my_task_id_1', 'my_task_id_2'])

What I need is would possibly look like this:

# pulls multiple values but from a single
pulled_value_1, pulled_value_2 = ti.xcom_pull(key=['my_key_1', 'my_key_2'], task_ids='my_task_id')

I can't find this in the documentation.
Is this even possible?
If yes, it makes a single database query in the background, or just repeats a single query multiple times? If not, how could I get similar behavior?

Upvotes: 2

Views: 7422

Answers (1)

Elad Kalif
Elad Kalif

Reputation: 16079

Answering your questions:

  1. There is no such feature. xcom_pull accepts task_ids: Optional[Union[str, Iterable[str]]] but with the same key. You can open a PR to Airflow for adding the functionality you seek.

  2. As for number of queries: I assume that by "repeats a single query" you are asking if it execute a query per task_id. The answer is No. Airflow did this optimization in PR. In the source code you can see that xcom_pull() uses Xcom.get_many() and get_many creates a filter using IN which allows to compare against multiple values. You can see the relevant code lines here.

Upvotes: 3

Related Questions