CODEWITHSUNDEEP
fileshellfilterdirectoryscp
derrdji
derrdji

Reputation: 13301

How to filter files when using scp to copy dir recursively?

I need to copy all the .class files from server to local with all dir reserved. e.g. server:/usr/some/unknown/number/of/sub/folders/me.class will be /usr/project/backup/some/unknown/number/of/sub/folders/me.class the problem is, there are many other useless files such as .svn-base files that i don't want. how can i filter them so I only scp .class files?

Upvotes: 122

Views: 165293

Answers (10)

Ayoade David
Ayoade David

Reputation: 1

If you're using rsync and need to exclude specific files or directories, you can use the --exclude flag. However, if the list of files or directories you want to exclude is large, it's more efficient to put all the exclude patterns in a file and use the --exclude-from option to reference that file.

For example, instead of using multiple --exclude options like this:

rsync -a --exclude='file1.txt' --exclude='dir1/*' --exclude='dir2' src_directory/ dst_directory/

You can create a file, say .gitignore, that contains all the exclude patterns

rsync -a --exclude-from='.gitignore' src_directory/ dst_directory/

Upvotes: -1

Vinay Babu
Vinay Babu

Reputation: 1

scp -i /home/<user>/.ssh/id_rsa -o "StrictHostKeyChecking=no" -rp /source/directory/path/[!.]* <target_user>@<target_system:/destination/directory/path

Upvotes: 0

rgtk
rgtk

Reputation: 3490

To exclude dotfiles in base directory:

scp -r [!.]* server:/path/to/something

[!.]* is a shell glob that expands to all files in working directory not starting with a dot.

Upvotes: 102

Scott Chu
Scott Chu

Reputation: 992

If you indeed wanna use scp, there's a indirect way.Say we want to copy all .jpg file under local folder '/src' to folder '/dst' in remote server 10.1.1.2:

#make a clean temp folder
mkdir /tmp/ttt
#copy all .jpg file and retain folder structure as-is
find /src -type f -name *.jpg -exec cp --parents \{\} /tmp/ttt \;
#copy to remote target folder as-is and retain original time attributes
scp -rp /tmp/ttt/* 10.1.1.2:/dst
#if copy ok, remove temp folder
rm -rf /tmp/ttt

Upvotes: 1

Suneet Khurana
Suneet Khurana

Reputation: 449

Below command for files.

scp `find . -maxdepth 1 -name "*.log" \! -name "hs_err_pid2801.log" -type f` root@IP:/tmp/test/

  1. IP will be destination server IP address.
  2. -name "*.log" for include files.
  3. \! -name "hs_err_pid2801.log" for exclude files.
  4. . is current working dir.
  5. -type f for file type.

Below command for directory.

scp -r `find . -maxdepth 1 -name "lo*" \! -name "localhost" -type d` root@IP:/tmp/test/

you can customize above command as per your requirement.

Upvotes: 1

varun
varun

Reputation: 361

With ssh key based authentication enabled, the following script would work.

for x in `ssh user@remotehost 'find /usr/some -type f -name *.class'`; do y=$(echo $x|sed 's/.[^/]*$//'|sed "s/^\/usr//"); mkdir -p /usr/project/backup$y; scp $(echo 'user@remotehost:'$x) /usr/project/backup$y/; done

Upvotes: 1

Gavin Gilmour
Gavin Gilmour

Reputation: 6963

I'd probably recommend using something like rsync for this due to its include and exclude flags, e.g:-

rsync -rav -e ssh --include '*/' --include='*.class' --exclude='*' \
server:/usr/some/unknown/number/of/sub/folders/ \ 
/usr/project/backup/some/unknown/number/of/sub/folders/

Some other useful flags:

  • -r for recursive
  • -a for archive (mostly all files)
  • -v for verbose output
  • -e to specify ssh instead of the default (which should be ssh, actually)

Upvotes: 176

Ville Laurikari
Ville Laurikari

Reputation: 29248

There is no feature in scp to filter files. For "advanced" stuff like this, I recommend using rsync:

rsync -av --exclude '*.svn' user@server:/my/dir .

(this line copy rsync from distant folder to current one)

Recent versions of rsync tunnel over an ssh connection automatically by default.

Upvotes: 51

nik
nik

Reputation: 13450

Since you can scp you should be ok to ssh,
either script the following or login and execute...

# After reaching the server of interest
cd /usr/some/unknown/number/of/sub/folders
tar cfj pack.tar.bz2 $(find . -type f -name *.class)

return back (logout) to local server and scp,

# from the local machine
cd /usr/project/backup/some/unknown/number/of/sub/folders
scp you@server:/usr/some/unknown/number/of/sub/folders/pack.tar.bz2 .
tar xfj pack.tar.bz2

If you find the $(find ...) is too long for your tar change to,

find . -type f -name *.class | xargs tar cfj pack.tar.bz2

Finally, since you are keeping it in /usr/project/backup/,
why bother extraction? Just keep the tar.bz2, with maybe a date+time stamp.

Upvotes: 11

dfa
dfa

Reputation: 116304

  1. Copy your source folder to somedir:

    cp -r srcdir somedir

  2. Remove all unneeded files:

    find somedir -name '.svn' -exec rm -rf {} \+

  3. launch scp from somedir

Upvotes: -4

Related Questions