CODEWITHSUNDEEP
hadoopexportteradatasqoop
ae8
ae8

Reputation: 53

Sqoop export for 100 million records faster

I have query similar to the below

sqoop export 
    --connect jdbc:teradata://server/database=BIGDATA 
    --username dbuser 
    --password dbpw 
    -Dsqoop.export.records.per.statement=500
    --batch 
    --hive-table country
    --table COUNTRY
    --input-null-non-string '\\N' \ --input-null-string '\\N'

The above query is working fine for 3 million records(Takes 1 hour to load the data to TeraData table). For 100 million records to export the data to Teradata empty table i think it may take more time to complete the export. How can i efficiently write the query to export the data faster without failing the query?

Upvotes: 1

Views: 3298

Answers (1)

Laenka-Oss
Laenka-Oss

Reputation: 994

You may want to consider increasing your --fetch-size (the number of entries that sqoop has to fetch per scoop of data) from the default 1000 to e.g --fetch-size 10000 0r 20000 depending on your available memory as well as your environments' bandwidth.

Upvotes: 1

Related Questions