CODEWITHSUNDEEP
google-bigquery
SMAnalytics
SMAnalytics

Reputation: 1

Merging 2 partitioned tables BigQuery

I am trying to merge 2 partitioned tables in BigQuery:

I want to get data from last partition of source table and merge it to target table. To filter the search task on tagret table I need to use the field 'date' from the data of source table. I wrote a query but editor shows following query error:

Cannot query over table 'MyDataSet.target_t' without a filter over column(s) 'date'

Here is my query:

declare latest default (select date(max(_PARTITIONTIME)) as latest from MyDataSet.source_t where _PARTITIONTIME >= timestamp(date_sub(current_date(),interval 1 day))); 
declare first_date default (select min(date) as first_date from MyDataSet.source_t where date(_PARTITIONTIME) = latest);
merge `MyDataSet.target_t` as a
using (select * from `MyDataSet.source_t` where _PARTITIONTIME = latest) as b 
on
  a.date >= first_date and
  a.date = b.date and
  a.account_id = b.account_id and 
  a.campaign_id = b.campaign_id and 
  a.adset_id = b.adset_id and 
  a.ad_id = b.ad_id 
when matched then update set 
  a.account_name = b.account_name, 
  a.campaign_name = b.campaign_name, 
  a.adset_name = b.adset_name, 
  a.ad_name = b.ad_name, 
  a.impressions = b.impressions, 
  a.clicks = b.clicks, 
  a.spend = b.spend, 
  a.date = b.date 
when not matched then insert row;

If I input date instead of 'latest' variable ("where _PARTITIONTIME = '2020-10-01') as b") there wont be any error. But I want to filter the source table properly. And I don't get it how it affects the following 'on' statement and why everything brokes >.< Could you please help? What is a proper syntax to write such query. And is there any other ways to run such merge without variables?

Upvotes: 0

Views: 1219

Answers (1)

Yun Zhang
Yun Zhang

Reputation: 5503

declare latest timestamp;

Your variable latest is a TIMESTAMP. Making it a DATE type then your query should work.

------ Update --------

The error is complaining about MyDataSet.target_t doesn't have a good filter on date column. Could you try put after on clause a.date = latest (if this is not the right filter, come up with other constant filter)

Upvotes: 0

Related Questions