CODEWITHSUNDEEP
hadoopoutputhdfssqoop
Chhaya Vishwakarma
Chhaya Vishwakarma

Reputation: 1437

How to give custom name to Sqoop output files

When I import data to hive using sqoop bydefault it creates file name as part-m-0000, part-m-0001 etc on HDFS.

Is it possible to rename these files?

If i wish to give some meaningfull name like suffxing file name with date to indicate load how can I do it?

Please suggest

Upvotes: 2

Views: 3459

Answers (2)

54l3d
54l3d

Reputation: 3973

Yes we can ! see here

sqoop import -D mapreduce.output.basename=`date +%Y-%m-%d`

Upvotes: 1

Nadine
Nadine

Reputation: 1638

You can't do it with sqoop directly, but you can rename them in HDFS after sqoop is done importing:

today=`date +%Y-%m-%d`
files=$(hadoop fs -ls /path-to-files | awk  '{print $8}')
for f in $files; do hadoop fs -mv $f $f$today; done

The first command gets today's date. The second command gets all the filenames within your directory. The third command renames those files, appending the date.

Upvotes: 1

Related Questions