CODEWITHSUNDEEP
snowflake-cloud-data-platformdata-modelingdbt
DSA
DSA

Reputation: 780

What is the real use of DBT (data build tool) & Snowflake together?

DBT is T in ELT, that's what I see in all the articles, blogs, and videos. I understand the concept of modeling in DBT in ways like whatever data modeling logic we create can be stored as a version for future reference. But I still have a few questions,

  1. Is DBT only meant to store data modeling logic versions (along with model testing, preview, and all)?
  2. If I'm creating continuous data transformation pipeline in snowflake, how DBT will help there (apart from creating a model and model constraint check)?

I understand real usage this way,

  1. Use DBT model to create table/views in snowflake (which will help to version the changes).
  2. Use Snowflake streams & tasks to continuously flow the data transformation and to create a mart.

Any expert thought?

Upvotes: 0

Views: 3802

Answers (1)

Lukasz Szozda
Lukasz Szozda

Reputation: 175726

The project has two components:

  1. dbt model - creation of views and tables
  2. dbt run - run of the model hierarchy/DAG

Data ingestion is performed continuously via 3rd party tools.

As I understand the goal is to keep the model as-is within dbt but be able to run it as soon as possible when new data arrives to have it "fresh" for the end user.

Possibilities:

a) use only view materializations in dbt in the entire flow. Views are evaluated during runtime so the new data will be available when it arrives to source/base table

b) schedule the dbt run for critical path(--select) in more frequent manner, effectively introducing micro-batching

c) wait for the new Snowflake feature called dynamic tables(previously known as "materialized tables") and dbt support to materialize as "dynamic tables"

CREATE [ OR REPLACE ] DYNAMIC TABLE 
  LAG = ' { seconds | minutes | hours | days }'
  WAREHOUSE = 
  AS SELECT

More at:

See how Snowflake is removing the boundaries between streaming and batch systems by providing native streaming pipeline capabilities. You'll learn about the latest features such as Snowpipe Streaming and Materialized Tables.

Upvotes: 3

Related Questions