CODEWITHSUNDEEP
hadoopimporthdfsrdbmssqoop
Venu A Positive
Venu A Positive

Reputation: 3062

How to import only new data by using Sqoop?

Let me give an example: I exported 1TB of data yesterday. Today, the database got another 1GB of data. If I try to import the data again today, Sqoop will import 1TB+1GB of data, then I am merging it. So it's a headache. I want to import only new data and append it to the old data. In this way, on a daily basis, I'll pull the RDBMS data into HDFS.

Upvotes: 2

Views: 2187

Answers (1)

Sandeep Singh
Sandeep Singh

Reputation: 7990

You can use sqoop Incremental Imports:

Sqoop provides an incremental import mode which can be used to retrieve only rows newer than some previously-imported set of rows.

Incremental import arguments:

--check-column (col) Specifies the column to be examined when determining which rows to import. --incremental (mode) Specifies how Sqoop determines which rows are new. Legal values for mode include append and last modified.

--last-value (value) Specifies the maximum value of the check column from the previous import.

Reference: https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_incremental_imports

For Incremental Import: You would need to specify a value in a check column against a reference value for the most recent import. For example, if the –incremental append argument was specified, along with –check-column id and –last-value 100, all rows with id > 100 will be imported. If an incremental import is run from the command line, the value which should be specified as –last-value in a subsequent incremental import will be printed to the screen for your reference. If an incremental import is run from a saved job, this value will be retained in the saved job. Subsequent runs of sqoop job –exec some Incremental Job will continue to import only newer rows than those previously imported.

For importing all the tables at one go, you would need to use sqoop-import-all-tables command, but this command must satisfy the below criteria to work

Each table must have a single-column primary key. You must intend to import all columns of each table. You must not intend to use non-default splitting column, nor impose any conditions via a WHERE clause.

Reference: https://hortonworks.com/community/forums/topic/sqoop-incremental-import/

Upvotes: 4

Related Questions