CODEWITHSUNDEEP
hivejinja2airflow
RGregg
RGregg

Reputation: 135

Jinja :Undefined error when calling in Hive (Airflow)

I'm setting up my DAG in Airflow (GC Composer) and trying to use Jinja2 templates in my DataProcHiveOperator. But I'm getting a jinja2.exceptions.UndefinedError: 'jinja' is undefined error. However, nowhere in my code I'm calling 'jinja'. What am I missing?

I've already tried many suggestions on other Stackoverflow similar questions, but none of them worked (maybe because of something I'm missing). I've tried to create an object called 'jinja' in my DAG file, but also didn't work.

This is my DAG

from airflow.models import DAG, Variable
from airflow.contrib.operators.dataproc_operator import DataProcHiveOperator, DataprocClusterScaleOperator
from airflow.utils.trigger_rule import TriggerRule
from datetime import datetime, timedelta, time

execution_date = "{{ ds }}"

args = {
    'owner': 'Raul Gregg',
    'start_date': datetime(2018,12,12),
    'provide_context': True,
    'depends_on_past': False,
    'retries': 0,
    'retry_delay': timedelta(seconds=30),
    'project_id': Variable.get('gcp_project'),
    'cluster_name': 'hive-cluster',
    'region': 'europe-west3'
}

dag = DAG('dag_testing', 
    default_args=args,
    max_active_runs=9,
    schedule_interval=timedelta(days=1)
    )

test_dim_facets = DataProcHiveOperator(
    task_id='test_dim_facets',
    query='/hql/tests/tests_dim_facets_1.q',
    dag=dag
    )

And here is the .q file, which isn't even called by the DAG due to the error above;

SELECT 
    distinct original_color_flag,
    {{ execution_date }} as exec_date

from omni_offer.dim_color_flagged_models;

All I want to do is to run a query with a Jinja2 date template. Simple like that, but really hard to achieve. Thanks!

Upvotes: 0

Views: 2347

Answers (1)

RGregg
RGregg

Reputation: 135

I managed to find the problem with my own code. For the DataProcHiveOperator to work properly, it is necessary to create an object called 'jinja' with the dictionary of keywords you are going to use. Here is the snippet I added to my .py code:

jinja = {
    """
    'ds': '{{ ds }}',
    'ds_nodash': '{{ ds_nodash }}'
    """
}

Now, if I call those items in my .q file, it will read and substitute the text accordingly.

Upvotes: 0

Related Questions