CODEWITHSUNDEEP
jinja2dbt
jeffgrills
jeffgrills

Reputation: 87

How do make it so that every time the incremental load in dbt runs it just updates the new rows from when it last ran?

How do make it so that every time the incremental load in dbt runs it just updates the new rows from when it last ran?

{% if is_incremental() %}

    /* code */

    {% endif %}

Upvotes: 2

Views: 7074

Answers (1)

sgdata
sgdata

Reputation: 2763

Your two main resources for doing this are going to be in the dbt docs already:

  1. How to build incremental models in dbt: https://docs.getdbt.com/docs/building-a-dbt-project/building-models/configuring-incremental-models/

  2. Incremental models particular to bigquery: https://docs.getdbt.com/reference/resource-configs/bigquery-configs/#merge-behavior-incremental-models

Most likely the model will look something like:

{{
    config(
        materialized='incremental'
    )
}}

select <columns>
from <my_table>
{% if is_incremental() %}
  where <my_table>.<record_update_timestamp> >= (
      select max(<my_table>.<record_update_timestamp>) from {{ this }}
  )
  {% endif %}

Full example in from docs in: https://docs.getdbt.com/docs/building-a-dbt-project/building-models/configuring-incremental-models/#defining-a-uniqueness-constraint-optional

Upvotes: 6

Related Questions