CODEWITHSUNDEEP
pythonjinja2airflow
Julian Larralde
Julian Larralde

Reputation: 9

Airflow - How can I access an execution parameter in a non templated field?

I've managed to retrieve the execution parameter in my Airflow 1.10.15 dag but only under templated parameters (like PythonOperator and BashOperator) as described here

I need to use the execution parameter in an operator field that is not templated, namely GoogleCloudStorageToBigQueryOperator - field_delimiter and a few others.

Is it possible at all? Any Alternatives?

Thanks

Upvotes: 1

Views: 425

Answers (1)

Elad Kalif
Elad Kalif

Reputation: 15931

You can create your own operator with the same behavior that adds namespace to template_fields:

class MyGoogleCloudStorageToBigQueryOperator(GoogleCloudStorageToBigQueryOperator):
         template_fields = GoogleCloudStorageToBigQueryOperator.template_fields +('field_delimiter',)

Then simply use MyGoogleCloudStorageToBigQueryOperator as you are used to. It has all the abilities of GoogleCloudStorageToBigQueryOperator with additional functionality of field_delimiter as templated field.

Note: GoogleCloudStorageToBigQueryOperator is deprecated. You should import GCSToBigQueryOperator from google backport provider.

Upvotes: 1

Related Questions