smarinova1
smarinova1

Reputation: 49

Error unrecognized argument --hive-partition-key

I am getting error Unrecognized argument --hive-partition-key , when I run the following statement:

sqoop import 
--connect 'jdbc:sqlserver://192.168.56.1;database=xyz_dms_cust_100;username-hadoop;password=hadoop' 
--table e_purchase_category 
--hive_import 
--delete-target-dir 
--hive-table purchase_category_p 
--hive-partition-key "creation_date" 
--hive-partition-value "2015-02-02"

The partitioned table exists.

Upvotes: 2

Views: 2044

Answers (2)

Akash Saini
Akash Saini

Reputation: 111

It could applied too :

sqoop import --connect jdbc:mysql://localhost/akash
--username root
--P
--table mytest
--where "dob='2019-12-28'"
--columns "id,name,salary"
--target-dir /user/cloudera/
--m 1 --hive-table mytest
--hive-import
--hive-overwrite
--hive-partition-key dob
--hive-partition-value '2019-12-28'

Upvotes: 0

Sai Neelakantam
Sai Neelakantam

Reputation: 939

Hive partition key (creation_date in your example) should not be part of your database table when you are using hive-import. When you are trying to create table in hive with partition you will not include partition column in your table schema. The same applies to sqoop hive-import as well.

Based on your sqoop command, i am guessing that creation_date column is present in your SQLServer table. If yes, you might be getting this error

ERROR tool.ImportTool: Imported Failed: 
Partition key creation_date cannot be a column to import.

To resolve this issue, i have two solutions:

  1. Make sure that the partition column is not present in the SQLServer table. So, when sqoop creates hive table it includes that partition column and its value as directory in hive warehouse.
  2. Change the sqoop command by including a free form query to get all the columns expect the partiton column and do hive-import. Below is a example for this solution

Example:

sqoop import 
--connect jdbc:mysql://localhost:3306/hadoopexamples 
--query 'select City.ID, City.Name, City.District, City.Population from City where $CONDITIONS' 
--target-dir /user/XXXX/City
--delete-target-dir 
--hive-import 
--hive-table City
--hive-partition-key "CountryCode" 
--hive-partition-value "USA" 
--fields-terminated-by ',' 
-m 1

Another method: You can also try to do your tasks in different steps:

  1. Create a partition table in hive (Example: city_partition)
  2. Load data from RDBMS to sqoop using hive-import into a plain hive table (Example: city)
  3. Using insert overwrite, import data into partition table (city_partition) from plain hive table (city) like:

INSERT OVERWRITE TABLE city_partition PARTITION (CountryCode='USA') SELECT id, name, district, population FROM city;

Upvotes: 1

Related Questions