CODEWITHSUNDEEP
hadoopcopyhdfssubdirectorycp
R. Sluiter
R. Sluiter

Reputation: 172

hdfs copy multiple files to same target directory

I learned that if you want to copy multiple files from one hadoop folder to another hadoop folder you can better create one big 'hdfs dfs -cp' statement with lots of components, instead of creating several hdfs dfs -cp statements. With 'better' I mean that it will improve the overal time it takes to copy files: one command is quicker than several separate -cp commands run after each other.

When I do this and my target directory is the same for all files that I want to copy I get a warning.

I'm executing the following command:

hdfs dfs -cp -f /path1/file1 /pathx/target /path2/file2 /pathx/target /path3/file3 /pathx/target

After executing it I get the following warning returned:

cp: `/pathx/target' to `/pathx/target/target': is a subdirectory of itself

Although I get this weird warning the copy itself succeeds like it should. Is this a bug or am I missing something?

Upvotes: 6

Views: 19561

Answers (3)

Shiva Ambati
Shiva Ambati

Reputation: 1

-cp did not worked for me... I used the following and it worked

hdfs dfs -put /path1/file1 /path2/file2 path3/file3 /pathx/target

Upvotes: 0

Abhi
Abhi

Reputation: 21

Or You could do it like this:

hadoop fs -cp /path1/{file1, file2, file3} /pathx/target

If you want to copy all the files then:

hadoop fs -cp /path1/* /pathx/target

Upvotes: 1

Luca Natali
Luca Natali

Reputation: 414

Try to use the following syntax:

hadoop fs -cp /path1/file1 /path2/file2 path3/file3 /pathx/target

Upvotes: 14

Related Questions