CODEWITHSUNDEEP
airflowairflow-2.x
Stephen
Stephen

Reputation: 8860

Use Smart Sensors and still get context variable

I am on Airflow 2.1.4 and I am trying to modify a custom sensor to act as a Smart Sensor.

Among other things, to allow a custom sensor to work as a Smart Sensor you need to give it a poke_context_fields class variable. This isn't very well documented, but I think it's just a list of the arguments to __init__ that you want to also be passed to self.poke() when that is called by the Smart Sensor DAG/Shard (though I could be wrong).

So I have it like:

poke_context_fields = ['myarg1', 'myarg2']

I have tested this out, but it seems there is a problem: When the Smart Sensor DAG calls self.poke(), it forwards along those arguments as expected, but it does NOT give me the regular context variable that my poke method expects. Unfortunately my code won't work without access to that variable, because it needs certain properties like context['ds'], context['task_instance'], etc, which would only be available at execution time and not when Python parses the class variable.

I have read the following (https://github.com/apache/airflow/issues/11893) but I don't 100% follow. Is there a workaround for this or should I just conclude that I can't use Smart Sensors and should wait to use Deferrable Operators which was released in 2.2.0?

Upvotes: 0

Views: 607

Answers (1)

Elad Kalif
Elad Kalif

Reputation: 15979

Smart Sensors is/was an experimental feature. Eventually it was decided to deprecate it (see PR) starting from version 2.3.0 in favor of Deferrable Operators.

To quote the docs:

Deferrable Operators essentially supersede Smart Sensors, and should be preferred for almost all situations.

Upvotes: 1

Related Questions