CODEWITHSUNDEEP
hadoopdistcp
Bhavesh
Bhavesh

Reputation: 919

hadoop discp issue while copying singe file

(Note: I need to use distcp to get parallelism)

I have 2 files in /user/bhavesh folder

enter image description here

I have 1 file in /user/bhavesh1 folder

enter image description here

Copying 2 files from /user/bhavesh to /user/uday folder (This work fine)

enter image description here

enter image description here

This create /user/uday folder

Copying 1 file from /user/bhavesh1 to /user/uday1 folder if creates file instead of folder

enter image description here

enter image description here

What i need is if there is one file /user/bhavesh1/emp1.csv i need is it should create /user/uday1/emp1.csv [uday1 should form as directory] Any suggestion or help is highly appreciated.

Upvotes: 1

Views: 2171

Answers (1)

SachinJose
SachinJose

Reputation: 8522

In unix systems, when u copy a single file by giving destination directory name ending with /user/uday1/, destination directory will be created, however hadoop fs -cp command will fail if destination directory is missing.

When it comes it hdfs distcp, file/dir names ending with / will be ignored if it's a single file. One workaround is to create the destination directory before executing distcp command. you may add -p option in -mkdir to avoid directory already exists error. 

hadoop fs -mkdir -p /user/uday1  ; hadoop distcp /user/bhavesh1/emp*.csv /user/uday1/

this works for both single file and multiple files in the source directory.

Upvotes: 1

Related Questions