CODEWITHSUNDEEP
hadoopsqoopdata-migration
Rj1
Rj1

Reputation: 469

How to specify multiple columns for incremental data in Sqoop?

I am using following query to fetch incremental data in sqoop-

bin/sqoop job --create JOB_NAME -- import  --connect jdbc:oracle:thin:/system@HOST:PORT:ORACLE_SERVICE --username USERNAME --password-file /PASSWORD_FILE.txt --fields-terminated-by ',' --enclosed-by '"'  --table SCHEMA.TABLE_NAME --target-dir /TARGET_DIR -m 2 --incremental append --check-column NVL(UPDATE_DATE,INSERT_DATE) --last-value '2019-01-01 00:00:00.000'  --split-by PRIMARY_KEY --direct

It throwing error for Multiple columns in --check-columns parameters.

Is there any approcach to specify multi columns in --check-column parameter?

I want to fetch data , if UPDATE_DATE field contains null value then it should fetch the data on the basis of INSERT_DATE column.

I want to extract transaction records from a table which is being updated daily , and if the records is inserted first time then there is no value in UPDATED_DATE column. That's why I need to compare both columns while fetching data from table.

Any help regarding this would be highly appreciated.

Upvotes: 1

Views: 566

Answers (1)

Vin
Vin

Reputation: 525

As per my understanding it doesn't look like it's possible to have 2 check columns when doing incremental imports, so the only way we can manage to get this done is with 2 separate imports:

  1. Incremental import with the Insert date as check column for first time records
  2. Incremental import with the updated column as check column for UPDATED records

Upvotes: 1

Related Questions