CODEWITHSUNDEEP
sqlhivehiveql
baha-kev
baha-kev

Reputation: 3059

Combine many tables in Hive using UNION ALL?

I'm trying to append one variable from several tables together (aka row-bind, concatenate) to make one longer table with a single column in Hive. I think this is possible using UNION ALL based on this question ( HiveQL UNION ALL ), but I'm not sure an efficient way to accomplish this?

The pseudocode would look something like this:

CREATE TABLE tmp_combined AS
SELECT b.var1 FROM tmp_table1 b
UNION ALL
SELECT c.var1 FROM tmp_table2 c
UNION ALL
SELECT d.var1 FROM tmp_table3 d
UNION ALL
SELECT e.var1 FROM tmp_table4 e
UNION ALL
SELECT f.var1 FROM tmp_table5 f
UNION ALL
SELECT g.var1 FROM tmp_table6 g
UNION ALL
SELECT h.var1 FROM tmp_table7 h;

Any help is appreciated!

Upvotes: 9

Views: 86411

Answers (3)

Haoyan
Haoyan

Reputation: 51

I would say that's both straightforward and efficient way to do the row-bind, at least, that's what I would use in my code. Btw, it might cause you some syntax error if you put your pseudo code directly, you may try:

create table join_table as
select * from
(select ...
join all
select
join all
select...) tmp;

Upvotes: 4

Marimuthu Kandasamy
Marimuthu Kandasamy

Reputation: 516

Try with following coding...

Select * into tmp_combined  from 
(
    SELECT b.var1 FROM tmp_table1 b
    UNION ALL
    SELECT c.var1 FROM tmp_table2 c
    UNION ALL
    SELECT d.var1 FROM tmp_table3 d
    UNION ALL
    SELECT e.var1 FROM tmp_table4 e
    UNION ALL
    SELECT f.var1 FROM tmp_table5 f
    UNION ALL
    SELECT g.var1 FROM tmp_table6 g
    UNION ALL
    SELECT h.var1 FROM tmp_table7 h
) CombinedTable

Use with the statement : set hive.exec.parallel=true

This will execute different selects simultaneously otherwise it would be step by step.

Upvotes: 34

ChikuMiku
ChikuMiku

Reputation: 509

I did same concept but for different tables employee and location that might help you I believe :

DATA:Table_e-employee
empid empname
13  Josan
8   Alex
3   Ram
17  Babu
25  John

Table_l-location
empid emplocation
13  San Jose
8   Los Angeles
3   Pune,IN
17  Chennai,IN
39  Banglore,IN

hive> SELECT e.empid AS a ,e.empname AS b FROM employee e 
UNION ALL 
SELECT l.empid AS a,l.emplocation AS b FROM location l;

OutPut with alias a and b:

13  San Jose
8   Los Angeles
3   Pune,IN
17  Chennai,IN
39  Banglore,IN
13  Josan
8   Alex
3   Ram
17  Babu
25  John

Upvotes: 3

Related Questions