CODEWITHSUNDEEP
pythonjinja2airflow
Aravind Krishnakumar
Aravind Krishnakumar

Reputation: 2777

Airflow XCOM pull is not rendering

I have a custom operator where in the argument list I am using xcom_pull to get values from XCOM. But it is not rendering to actual value instead it remains as the string.

download= CustomSparkSubmitOperator(
    task_id='download',
    spark_args=command_func(
        env, application,
        '%s/spark_args' % key,
        ['--input_file', "{{ ti.xcom_pull('set_parameters', key='input_file') }}",
         '--output_file', "{{ ti.xcom_pull('set_parameters', key='output_file') }}"
         ],
    provide_context=True,
    dag=dag)

The operator returns the following output:

spark-submit --deploy-mode cluster ..... --input_file "{{ ti.xcom_pull('set_parameters', key='input_file') }}" --output_file "{{ ti.xcom_pull('set_parameters', key='output_file') }}"

Upvotes: 0

Views: 2038

Answers (1)

Aravind Krishnakumar
Aravind Krishnakumar

Reputation: 2777

I was able to fix the problem of XCOM values not rendering when using as an argument in my CustomSparkSubmitEMROperator. Internally the operator inherits the EMROperators. For example

class CustomSparkSubmitEMROperator(EmrAddStepsOperator, EmrStepSensor):

So I needed to add the below template_fields as shown below

template_fields = ('job_flow_id', 'steps')

After adding the above lines the XCOM values where properly rendered and was able to see the correct values in the resultant spark-submit command

Upvotes: 1

Related Questions