CODEWITHSUNDEEP
sqlhadoophivehqlhiveql
elaspog
elaspog

Reputation: 1709

How to insert from a select query with dynamic partitioning on a column in Hive?

I'm trying to insert a computed partition. The partition's value needs to be computed from a key column. Assume that key_2 in the example always has 10 characters. The last 3 character I want to use in partition. I need dynamic partitioning.

My table is similar to this:

DROP TABLE exampledb.exampletable;
CREATE TABLE exampledb.exampletable (
    key_1 STRING,
    key_2 STRING,
    col_1 STRING,
    col_2 STRING
)
PARTITIONED BY (my_part STRING)
ROW FORMAT DELIMITED
  FIELDS TERMINATED BY '\u0001'
;

I've tried multiple variants of the HQL below:

INSERT OVERWRITE TABLE exampledb.exampletable
PARTITION(my_part)
SELECT 
    key_1,
    key_2,
    col_1,
    col_2,
    SUBSTR(key_2, -3) as my_part    -- not sure how to insert partition
FROM exampledb.exampletable_temp;

I couldn't figure out what's the correct solution for this. I always get a syntax error.

Does enyone know the solution for this? Thanks

UPDATE:

FAILED: SemanticException Partition spec {my_part=null} contains non-partition columns

UPDATE 2:

I've also tried tried to avoid NULL values by using this solution (as it was proposed below, but the error is the same):

INSERT OVERWRITE TABLE hvdb_as_aqua_guk_core.hvtb_aqua_guk_finding_mgn
PARTITION(my_part) ( key_1, key_2, col_1, col_2, my_part    )
SELECT 
        key_1,
        key_2,
        col_1,
        col_2,
        SUBSTR(key_2, -3) as my_part    -- not sure how to insert partition
FROM hvdb_as_aqua_guk_core.hvtb_aqua_guk_finding_mgn_temp2
WHERE key_2 IS NOT NULL
    AND SUBSTR(key_2, -3) IS NOT NULL;

Upvotes: 1

Views: 3304

Answers (1)

Andrea
Andrea

Reputation: 4503

You should specify explicitly all the column names you are inserting into. For example, your command should be something like this:

INSERT OVERWRITE TABLE exampledb.exampletable
PARTITION(my_part)(key_1, key_2, col_1, col_2, my_part)
SELECT 
    key_1,
    key_2,
    col_1,
    col_2,
    SUBSTR(key_2, -3)
FROM exampledb.exampletable_temp;

This should work.

UPDATE

I tried to create a test case, and INSERT OVERWRITE doesn't seem to work, but INSERT INTO is working. A workaround could be to delete all data from the destination table with TRUNCATE TABLE exampledb.exampletable, or delete all data from a specific partition with TRUNCATE TABLE test6 PARTITION (my_part = '001');, then run an INSERT INTO:

INSERT INTO exampledb.exampletable
PARTITION(my_part)(key_1, key_2, col_1, col_2, my_part)
SELECT
    key_1,
    key_2,
    col_1,
    col_2,
    SUBSTR(key_2, -3)
FROM exampledb.exampletable_temp;

Upvotes: 1

Related Questions