CODEWITHSUNDEEP
hivesqoop
Rajendra B
Rajendra B

Reputation: 11

How to create single file while using sqoop import with multiple mappers

I want to import data from Mysql using sqoop import but my requirement is i want to use 4 mappers but it should create only one file in hdfs target directory is there is any way to do this ?

Upvotes: 0

Views: 2535

Answers (3)

Prateek Jaiswal
Prateek Jaiswal

Reputation: 1

you can use below sqoop command..!!

Suppose database name is prateekDB and table name is Emp...!!

sqoop import  --connect "jdbc:mysql://localhost:3306/prateekDB"  --username=root \
--password=data  --table Emp  --target-dir /SqoopImport  --split-by empno

Upvotes: 0

ozw1z5rd
ozw1z5rd

Reputation: 3208

Add this option to sqoop 

--num-mappers 1

the sqoop log shows:

Job Counters 
    Launched map tasks=1
    Other local map tasks=1

and finally on hdfs ONE file is created.

Upvotes: -1

Dev
Dev

Reputation: 13773

No. there is no option in sqoop to re-partition files into 1 file.

I don't think this should be a headache of sqoop.

You can do it easily using getmerge feature of hadoop. Example:

hadoop fs -getmerge /sqoop/target-dir/ /desired/local/output/file.txt

Here

  • /sqoop/target-dir is the target-dir of your sqoop command (directory containing all the part files).

  • desired/local/output/file.txt is the combined single file.

Upvotes: 2

Related Questions