CODEWITHSUNDEEP
google-bigqueryairflow
Sridhar Srinivasan
Sridhar Srinivasan

Reputation: 69

Include multiple SQL queries in one BigQueryInsertJobOperator Airflow

The syntax looks like this,

configuration={

          "query": {
                    "query": "{% include ['query_2.sql'] %}",
                    "useLegacySql": False,
 }

Is there any way I can do it?

"query": "{% include ['query_2.sql', 'query_3.sql', 'query_4.sql'] %}"

Upvotes: 0

Views: 1122

Answers (2)

Sridhar Srinivasan
Sridhar Srinivasan

Reputation: 69

configuration={

      "query": {
                "query": "{% for f in ['query_1.sql','query_2.sql'] %}"
                          "{% include f %}"
                              ";"
                          "{% endfor %}",
                "useLegacySql": False }

Upvotes: 2

Neon
Neon

Reputation: 41

You can create a stored procedure listing all the queries you need and then you call it like this:

        t1 = BigQueryInsertJobOperator(
        task_id="exec_sp",
        project_id=PROJECT_ID,
        configuration={
            "query": {
                "query": f"CALL `{PROJECT_ID}.{DATASET}.stored_procedure_name` ();", 
                "useLegacySql": False,
                "timeoutMs": 3600000,
            }
        }
        )

Upvotes: 0

Related Questions