CODEWITHSUNDEEP
hadoopimporthiveclouderasqoop
dreamer
dreamer

Reputation: 1069

Using Sqoop incremental import as chunk-wise

Is it really possible to import chunk-wise data through sqoop incremental import?

Say I have a table with rowid 1,2,3..... N (here N is 100) and now I want to import it as chunk. Like
1st import: 1,2,3.... 20
2nd import: 21,22,23.....40
last import: 81,82,83....100

I have read about the sqoop job with incremental import and also know the --last-value parameter but do not know how to pass the chunk size. For the above example, chunk size here is 20.

Upvotes: 0

Views: 1090

Answers (1)

dreamer
dreamer

Reputation: 1069

I ended up by writing a script which will modify the parameter file with new where clause after each successful sqoop run. I'm running both through Oozie coordinator. I wanted to use --boundary-query but it doesn't work with chunk. That's why I had to do this work-around. Details of this work-around can be found here:

http://tmusabbir.blogspot.com/2013/05/chunk-data-import-incremental-import-in.html

Upvotes: 1

Related Questions