CODEWITHSUNDEEP
testinggoogle-bigquerydbtnotnull
Semion Abramov
Semion Abramov

Reputation: 31

DBT generic test not_null - compiled as select * from the table

I'm using DBT generic tests to monitor my BQ DB.

While using the "not_null" test, it's compiled as :

select * from 'server.schema.table' where tested_column is null

In my case, selecting the column itself vs selecting all is ~70 Gb difference.

Is there any way to edit generic tests in the code?

Tried to read documentation.

I would expect the compiler to change the query to select the tested column and not ALL columns.

Upvotes: 3

Views: 734

Answers (2)

meril
meril

Reputation: 592

dbt test by default will only select the specific column for not_null test.

However, when you have --store-failures flag when running the test, dbt will query select * to store the failed rows. select * on huge tables will cost you lot when running frequently.

Upvotes: 0

Paddy Alton
Paddy Alton

Reputation: 2358

While there is some customisation you can do the generic tests are a one-size-fits-all way of doing things.

It's possible to define a 'custom generic test' to do what you want.

However, an alternative might be to look at various DBT packages that define a variety of generic tests. For example, DBT Expectations defines an expect_column_values_to_not_be_null test which extends the functionality of the basic not_null test. From the docs it sounds like it would do what you want.

Upvotes: 0

Related Questions