CODEWITHSUNDEEP
hadoophiveteradatasqoop
Chhaya Vishwakarma
Chhaya Vishwakarma

Reputation: 1437

Incremental loads in Sqoop

I have a table in Teradata which is loaded with new data on daily basis.

I need to import this data to Hive. I'm trying to use Sqoop but how should I do incremental load using Sqoop?

I checked incremental load options available in Sqoop

--check-col

This options expects only numeric and unique data whereas my Teradata table is not having such column

I'm using date column inn --spli-by

How can I do incremental imports to Hive in this case?

Please suggest

Upvotes: 0

Views: 8969

Answers (2)

santosh rachuri
santosh rachuri

Reputation: 1

use one more "--append " command after the check column so that , the new data will not be replaced and you can see it saves in a new file in hdfs without replacing the old file.

Upvotes: 0

Jeremy Beard
Jeremy Beard

Reputation: 2725

If you use --incremental lastmodified mode then your --check-column is a timestamp that does not need to be numeric or unique.

See: Sqoop incremental imports.

Upvotes: 2

Related Questions