CODEWITHSUNDEEP
centosrsynccentos6
Andron
Andron

Reputation: 6631

How to use Rsync to copy only specific subdirectories (same names in several directories)

I have such directories structure on server 1:

And I want duplicate this folder structure on server 2, but copy only directories/subdirectories of unique_folder1. I.e. as result must be:

I know that rsync is very good for this. I've tried 'include/exclude' options without success.

E.g. I've tried:

rsync -avzn --list-only --include '*/unique_folder1/**' --exclude '*' -e ssh [email protected]:/path/to/old/data/ /path/to/new/data/

But, as result, I don't see any files/directories:

receiving file list ... done
sent 43 bytes  received 21 bytes  42.67 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)

What's wrong? Ideas?

Additional information: I have sudo access to both servers. One idea I have - is to use find command and cpio together to copy to new directory with content I need and after that use Rsync. But this is very slow, there are a lot of files, etc.

Upvotes: 53

Views: 94318

Answers (4)

pip
pip

Reputation: 263

An alternative to Andron's Answer which is simpler to both understand and implement in many cases is to use the --files-from=FILE option. For the current problem,

rsync -arv --files-from='list.txt' old_path/data new_path/data

Where list.txt is simply

company1/unique_folder1/
company2/unique_folder1/
...

Note the -r flag must be included explicitly since --files-from turns off this behaviour of the -a flag. It also seems to me that the path construction is different from other rsync commands, in that company1/unique_folder1/ matches but /data/company1/unique_folder1/ does not.

Upvotes: 12

peterh
peterh

Reputation: 12573

For example, if you only want to sync target/classes/ and target/lib/ to a remote system, do

rsync -vaH --delete --delete-excluded --include='classes/***' --include='lib/***' \
      --exclude='*' target/ user@host:/deploy/path/

The important things to watch:

  • Don't forget the "/" from the end of the pathes, or you will get a copy into subdirectory.
  • The order of the --include, --exclude counts.
  • Contrary the other answers, starting with "/" an include/exclude parameter is unneeded, they will automatically appended to the source directory (target/ in the example).
  • To test, what exactly will happen, we can use a --dry-run flags, as the other answers say.
  • --delete-excluded will delete all content in the target directory, except the subdirectories we specifically included. It should be used wisely! On this reason, a --delete is not enough, it does not deletes the excluded files on the remote side by default (every other, yes), it should be given beside the ordinary --delete, again.

Upvotes: 4

Andron
Andron

Reputation: 6631

I've found the reason. As for me - it wasn't clear that Rsync works in this way.
So correct command (for company1 directory only) must be:

rsync -avzn --list-only --include 'company1/' --include 'company1/unique_folder1/***' --exclude '*' -e ssh [email protected]:/path/to/old/data/ /path/to/new/data

I.e. we need include each parent company directory. And of course we cannot write manually all these company directories in the command line, so we save the list into the file and use it.

Final things we need to do:

1.Generate include file on server 1, so its content will be (I've used ls and awk):

+ company1/  
+ company1/unique_folder1/***  
...  
+ companyN/  
+ companyN/unique_folder1/***

2.Copy include.txt to server 2 and use such command:

rsync -avzn                                        \
      --list-only                                  \
      --include-from '/path/to/new/include.txt'    \
      --exclude '*'                                \
      -e ssh [email protected]:/path/to/old/data/    \
      /path/to/new/data

Upvotes: 44

yonran
yonran

Reputation: 19134

If the first matching pattern excludes a directory, then all its descendants will never be traversed. When you want to include a deep directory e.g. company*/unique_folder1/** but exclude everything else *, you need to tell rsync to include all its ancestors too:

rsync -r -v --dry-run                       \
    --include='/'                           \
    --include='/company*/'                  \
    --include='/company*/unique_folder1/'   \
    --include='/company*/unique_folder1/**' \
    --exclude='*'

You can use bash’s brace expansion to save some typing. After brace expansion, the following command is exactly the same as the previous one:

rsync -r -v --dry-run --include=/{,'company*/'{,unique_folder1/{,'**'}}} --exclude='*'

Upvotes: 36

Related Questions