CODEWITHSUNDEEP
hadoophdfscloudera
roh
roh

Reputation: 1053

HDFS to HDFS move files force fully

As per my knowledge, there is no straight option to overwrite the file in HDFS while doing a move from one HDFS location to other, copying cp has the option to force. I'm trying to find if there is any hack to do that?

what we can do is hdfs dfs -cp -f /hdfs/location1 /hdfs/location2 but not hdfs dfs -mv -f /hdfs/location1/ /hdfs/location2/

One way to achieve my purpose is to do the hdfs dfs -cp -f /hdfs/location1 /hdfs/location2 first and then get rid of the location1 file with hdfs dfs -rm -r /hdfs/location1 but I don't want to do that for some reason. Any other approach with one single command will be appriciated.

thanks in advance !!

Upvotes: 4

Views: 10997

Answers (3)

Ani Menon
Ani Menon

Reputation: 28257

HDFS shell commands doesn't support a -f for mv.

You will have to do the copy then remove method (as stated by you).

Upvotes: 0

Sebastian Hätälä
Sebastian Hätälä

Reputation: 1045

Not a single command and rather a workaround:

For merging folders in HDFS I implemented a solution for this, since cp interacts with the physical data on the data nodes whereas mv only changes metadata on the name node.

for i in $(hdfs dfs -ls -R ${hdfs_src_dir} | grep "^-" | awk '{print $8}'); do
  i_dir=$(dirname "${i}")
  target_dir=${hdfs_tgt_dir}${i_dir#"$hdfs_src_dir"}
  hdfs dfs -mkdir -p $target_dir
  hdfs dfs -mv ${i} "${target_dir}"/ 2>&1
done

The first line selects all files in a destination recursively. The paths of each file in the target are then created and the files finally moved.

A downside is that the folder access rights are not moved but rather set at runtime.

Upvotes: 0

shaine
shaine

Reputation: 549

There is no way of doing this in a single command something like

hdfs dfs -rm {destination Uri} ; hdfs dfs -mv {source Uri} {destination Uri}

Would achieve this on a single line.

Upvotes: -4

Related Questions