CODEWITHSUNDEEP
databricksdbt
Yaakov Bressler
Yaakov Bressler

Reputation: 12128

DBT + Databricks -> Specify cluster for a test

When I run generic dbt commands (such as dbt debug), my project defaults to using the all_purpose_cluster defined in my connection profile.

When I execute dbt test --select a_big_model, my test utilizes the all_purpose_cluster which is too small for the computation – this causes my tests to fail because of resource issues. Is there a way for me to configure a specific cluster to be used when I run tests on a specific model?

Brainstorming:

Upvotes: 0

Views: 443

Answers (1)

Patrick
Patrick

Reputation: 56

You can override default configs for individual model files via config blocks.

In Databricks the http_path can be specified to switch the cluster used to run the model.

Note: For Databricks, this is only available in python models.

def model(dbt, _):
   dbt.config(
       materialized='table',
       http_path='...'
   )

Upvotes: 1

Related Questions