CODEWITHSUNDEEP
hivesqoop
Metadata
Metadata

Reputation: 2083

Sqoop incremental Import fail

I have the following table: mergetab in MYSQL.

id | name      | city        | rectime             |
---+-----------+-------------+---------------------+
 1 | Sidhartha | Hyderabad   | 2017-04-18 15:31:22 |
 2 | Saketh    | Bengaluru   | 2017-04-18 15:32:37 |
 3 | Sunny     | Mumbai      | 2017-04-18 15:32:57 |
 4 | Bobby     | Delhi       | 2017-04-18 15:33:15 |

I have inserted a record in it and it looks as below:

id | name      | city        | rectime             |
---+-----------+-------------+---------------------+
 1 | Sidhartha | Hyderabad   | 2017-04-18 15:31:22 |
 2 | Saketh    | Bengaluru   | 2017-04-18 15:32:37 |
 3 | Sunny     | Mumbai      | 2017-04-18 15:32:57 |
 4 | Bobby     | Delhi       | 2017-04-18 15:33:15 |
 5 | Madhavi   | Dharmavaram | 2017-04-18 16:57:09 |

Im trying to do a sqoop incremental import using the following command:

sqoop import --connect jdbc:mysql://127.0.0.1/mydb --table mergetab --username root --password cloudera --hive-import --hive-table mergetab --incremental-append --check-column id --last-value $(hive -S -e "select max(id) from mergetab");

Im getting the following error which I couldn't understand what is wrong with my sqoop command. Error:

17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Error parsing arguments for import:
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: WARN:
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: The
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: method
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: class
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: org.apache.commons.logging.impl.SLF4JLogFactory#release()
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: was
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: invoked.
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: WARN:
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: Please
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: see
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: http://www.slf4j.org/codes.html#release
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: for
17/04/18 17:35:26 ERROR tool.BaseSqoopTool: Unrecognized argument: an

Im practicing this in Cloudera VM. Can anyone let me what is the mistake that I am doing here ?

Upvotes: 0

Views: 492

Answers (1)

Pushkr
Pushkr

Reputation: 3619

The issue with your code line --last-value $(hive -S -e "select max(id) from mergetab") is, hive will always print header/logging information even in silent mode, so last-value will always get value like - 

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hive/2.1.0/libexec/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hadoop/2.7.3/libexec/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
5

you need to either write some twisted shell script to extract the last line or you can use beeline with showHeader and outputformat as shown below to just get the value directly assigned to variable

--last value $(beeline --showHeader=false --outputformat=tsv2 -e "your query")

Upvotes: 1

Related Questions