CODEWITHSUNDEEP
shellunixhadoop
shikha dwivedi
shikha dwivedi

Reputation: 15

creating a same directory structure in destination as in source using shell script

I'm loging to a remote server which has dir structure as follows-

cd /exp/gc/data/tmp/
ABCD
GED
TROOP

These directories in turn may or may not have subdirectories.My requirement is to copy files from remote server to hadoop(final destination) one at a time keeping the directory structure same as source.

eg: source:

cd /exp/gc/data/tmp/ABCD

now ABCD has 3 files and two subdirectories.So my target location(hadoop) should also have same directory structure i.e. ABCD must have 3 files and two subdir. Using rsync iam able to replicate the directory structure but directories with spaces or special character in the name are not getting copied. Code snippet: result = $(ssh username@hostname "find /path/ -type f") for file in $result do rsync -arsv username@hostname /path-source/ /target-path-tmp/ hadoop fs -copyFromLocal /target-path-tmp/ /hadoop -location/ rm -rf /target-path-tmp/*

Upvotes: 0

Views: 329

Answers (1)

vlp
vlp

Reputation: 603

I think what you are looking for is rsync, try it as follow:

    rsync -avz foo:src/bar /data/tmp

This would recursively transfer all files from the directory src/bar on the machine foo into the /data/tmp/bar directory on the local machine. The files are transferred in archive permissions, ownerships, etc. are preserved in the transfer. Additionally, compression will be used to reduce the size of data portions of the transfer.

example

Upvotes: 1

Related Questions