CODEWITHSUNDEEP
sqoopduplicate-data
maxmithun
maxmithun

Reputation: 1137

Partial and duplicate records while sqoop import

Sqoop import is resulting in duplicate/partial records when we are using the following setting

Verified the source data count say 1000 records

Verified the import data count say 1923 records

Upvotes: 3

Views: 2607

Answers (1)

maxmithun
maxmithun

Reputation: 1137

When using the split-by and field is non integer .

Sqoop uses TextSplitter which provides a warning as follows :

WARN db.TextSplitter: If your database sorts in a case-insensitive order, this may result in a partial import or duplicate records

WARN db.TextSplitter: You are strongly encouraged to choose an integral split column.
  • solution 1: use single mapper or 2
  • solution 2: use rank function in the query and use the --split-by on the rank field
  • solution 3: sort the --split-by field in ascending order in the query

Upvotes: 4

Related Questions