Using dbt_utils.union_relations wrong but don't know how/why?

Question

So I am a new DBT user, super cool stuff, but I am running into an issue with the union_relations macro. I am feeding relations to this function, but the compiled/run query is not finding any columns from the relations.

Here is the code I'm running:

{{dbt_utils.union_relations(relations=[ref('transform_hist_ca_map_stu_obj_assessment'), ref('transform_hist_sc_map_stu_obj_assessment')])}}


)
select *
from conformed_obj_assessment
where student_assessment_identifier is not null
and assessment_identifier is not null
and identification_code is not null
and student_unique_id is not null
and performance_level is not null

And this is the error I receive: syntax error at or near "from" LINE 1706: from __dbt__CTE__transform_hist_ca_map_stu_obj_a... ^ compiled SQL at target/run/rally_dw/conformed/conformed_student_objective_assessment.sql

Basically the first column is a DBT generated column, and there are supposed to be columns from the relations after that, but for some reason these columns are not being pulled in. I'm wondering if this is because the relations I want to pull from are currently ephemeral, so not materialized, So I'm wondering if that's causing an issue. Here is the compiled SQL, the CTEs return data, but for some reason it's not getting pulled into the last CTE.

  create  table "dashboarding"."dev_em_conformed"."conformed_student_objective_assessment__dbt_tmp"
  as (
    
with  __dbt__CTE__historical_ca_map_stu_obj_assessment as (

with hist_ca_map_stu_obj_assess as (
    select * from "dashboarding"."raw_ea"."historical_ca_map_student_obj_assessment"
),
cleaned as (
    select distinct
        source_org,
        assessment_id as assessment_identifier,
        student_assessment_identifier,
        student_unique_id,
        performance_levels  as performance_level,
        scale_score as score,
        assessment_id,
        to_date(test_date, 'YYYY-MM-DD') as test_date,
        identification_code,
        null as parent_objective_assessment_name
    from hist_ca_map_stu_obj_assess
)
select * from cleaned
),  __dbt__CTE__transform_hist_ca_map_stu_obj_assessment as (

with hist_ca_stu_obj_assess as (
    select * from __dbt__CTE__historical_ca_map_stu_obj_assessment
),
final as(
select
    null as source_org,
    student_assessment_identifier,
    assessment_id as assessment_identifier,
    identification_code as identification_code,
    null as school_year,
    student_unique_id,
    null as student_grade_level,
    null as assessment_grade_level,
    NULL as administration_date,
    null as administration_end_date,
    null as objective_assessment_name,
    score,
    performance_level,
    parent_objective_assessment_name,
    null as parent_objective_assessment_id
from hist_ca_stu_obj_assess

)
select * from final
), __dbt__CTE__historical_sc_map_stu_obj_assessment as (

with hist_sc_map_soa as (
    select * from "dashboarding"."raw_ea"."historical_sc_map_student_obj_assessment"
),
cleaned as (
    select distinct
        source_org,
        assessment_id as assessment_identifier,
        student_assessment_identifier,
        student_unique_id,
        performance_levels as performance_level,
        scale_score as score,
        assessment_id,
        to_date(test_date, 'YYYY-MM-DD') as test_date,
        identification_code,
        null as parent_objective_assessment_name
    from hist_sc_map_soa
)
select * from cleaned
),  __dbt__CTE__transform_hist_sc_map_stu_obj_assessment as (

with hist_sc_stu_obj_assess as (
    select * from __dbt__CTE__historical_sc_map_stu_obj_assessment
),
final as(
select
    null as source_org,
    student_assessment_identifier,
    assessment_id as assessment_identifier,
    identification_code as identification_code,
    null as school_year,
    student_unique_id,
    null as student_grade_level,
    null as assessment_grade_level,
    NULL as administration_date,
    null as administration_end_date,
    null as objective_assessment_name,
    score,
    performance_level,
    parent_objective_assessment_name,
    null as parent_objective_assessment_id
from hist_sc_stu_obj_assess

)
select * from final
),  conformed_obj_assessment as(



        (
            select

                cast('__dbt__CTE__transform_hist_ca_map_stu_obj_assessment' as 
    varchar
) as _dbt_source_relation,
                ---NO MORE COLUMNS???

            from __dbt__CTE__transform_hist_ca_map_stu_obj_assessment
        )

        union all
        

        (
            select

                cast('__dbt__CTE__transform_hist_sc_map_stu_obj_assessment' as 
    varchar
) as _dbt_source_relation,
                 ---NO MORE COLUMNS??

            from __dbt__CTE__transform_hist_sc_map_stu_obj_assessment
        )

        


)
select *
from conformed_obj_assessment
where student_assessment_identifier is not null
and assessment_identifier is not null
and identification_code is not null
and student_unique_id is not null
and performance_level is not null

  );

Any thoughts would be super appreciated thank you!

Answer 1

The union_relations macro relies on knowing what columns are in your relations (tables/views), as stored in the information schema. Since this model is ephemeral, there aren't any records in the information schema, which is why there's SQL like this:

select

cast('__dbt__CTE__transform_hist_ca_map_stu_obj_assessment' as 
    varchar
) as _dbt_source_relation,

from __dbt__CTE__transform_hist_ca_map_stu_obj_assessment

I noticed that you're using a slightly older version of dbt-utils — while we haven't fixed this issue, we have improved the way this issue is handled (released in v0.5.0).

A newer version of dbt-utils will helpfully tell you something like:

Compilation Error in model test_ephemeral (models/test_ephemeral.sql)

  The `union_relations` macro cannot be used with ephemeral models, as it relies on the information schema.

  `__dbt__CTE__my_ephemeral` is an ephemeral model. Consider making is a view or table instead.` is an ephemeral model. Consider making is a view or table instead.

So, as the (new) error message suggests — the only way around this is to make your upstream models a view or table.