CODEWITHSUNDEEP
sqlhadoophivecomparenosql
Jakub Zak
Jakub Zak

Reputation: 1232

How to compare two tables and return rows with difference with HIVE

So lets say I have a table with about 180 columns and 100 records. This table is backed up into temporary table and original one is removed. After this migration (change) is run on a pipeline which produces the same table. I want to compare the backed up table to the new one adn rows (records) with any difference to be moved to 3rd table (_result table) so I do:

INSERT OVERWRITE TABLE
  zakj_customers.customers_detail_result
SELECT
  acct_id, IF (a.title != b.title, 1, 0) title, IF (a.fname != b.fname, 1, 0) fname, IF (a.dob != b.dob, 1, 0) dob, IF (a.cr_date != b.cr_date, 1, 0) cr_date
FROM
  zakj_customers.customers_detail a
LEFT OUTER JOIN
  zakj_customers.customers_detail_backup b
ON
  (a.acct_id = b.acct_id)
ORDER BY 
  title DESC,fname DESC,dob DESC,cr_date DESC
HAVING
  title > 0 AND fname > 0 AND dob > 0 AND cr_date > 0
;

So oblivious this query is wrong, I'm not much into SQL, and I'm getting syntax errors, so I can't put it together right and on a ticket it was supplied in this format which is obviously wrong.

Can anyone see the way this could be done?

Cheers

Upvotes: 5

Views: 11940

Answers (1)

Peter Krassoi
Peter Krassoi

Reputation: 567

Must use "case when" instead of if:

Case When a.title <> b.title then 1 Else 0 End title

I wouldn't write having but the expression into the where condition:

INSERT Into
  zakj_customers.customers_detail_result
SELECT
  acct_id, a.title, a.fname, dob, a.cr_date
FROM
  zakj_customers.customers_detail a
LEFT OUTER JOIN
  zakj_customers.customers_detail_backup b
ON
  (a.acct_id = b.acct_id)
Where b.acct_id is null or a.title <> b.title or a.fname <> b.fname or a.cr_date <> b.cr_date;

"b.acct_id is null" is required to get the new records beacuse <> would filter out them.

(Ordering is totally unnecessary when inserting records.)

Upvotes: 3

Related Questions