CODEWITHSUNDEEP
hadoopexporthivesqoop
Kartik Voona
Kartik Voona

Reputation: 21

how to with deal primarykey while exporting data from Hive to rdbms using sqoop

Here is a my scenario i have a data in hive warehouse and i want to export this data into a table named "sample" of "test" database in mysql. What happens if one column is primary key in sample.test and and the data in hive(which we are exporting) is having duplicate values under that key ,then obviously the job will fail , so how could i handle this kind of scenario ?

Thanks in Advance

Upvotes: 0

Views: 1602

Answers (2)

Shasankar
Shasankar

Reputation: 692

If you want your mysql table to contain only the last row among the duplicates, you can use the following:

sqoop export --connect jdbc:mysql://<*ip*>/test -table sample --username root -P --export-dir /user/hive/warehouse/sample --update-key <*primary key column*> --update-mode allowinsert

While exporting, Sqoop converts each row into an insert statement by default. By specifying --update-key, each row can be converted into an update statement. However, if a particular row is not present for update, the row is skipped by default. This can be overridden by using --update-mode allowinsert, which allows such rows to be converted into insert statements.

Upvotes: 2

gbharat
gbharat

Reputation: 276

Beforing performing export operation ,massage your data by removing duplicates from primary key. Take distinct on that primary column and then export to mysql.

Upvotes: 0

Related Questions