CODEWITHSUNDEEP
findscp
Juan Pablo Barrios
Juan Pablo Barrios

Reputation: 493

Copy new files to remote host with FIND & SCP (or a better way that you may suggest)

I have a folder with tens of thousands of files and 520 GB of data that I sync every night by rsync to a remote host. This process usually takes around 5 hours. I would also like during the day to just copy those new and changed files to the remote location without comparing both trees, which is what rsync does, because sometimes people on the target would need to access the new files, so I started playing around with scp but I'm not sure how to solve two issues that I've encountered so far. Yes, I'm a newbie to bash. :)

This is my command that I would like to run every hour: find Folder/ -type f -mmin -60 -print0 -exec scp "/rootfolder1/subfolder1/{}" "user@host:/rootfolder2/subfolder2/{}" \;

The issues I'm having are: When the string passed to scp has a blank space it doesn't work and triggers the error "scp ambiguous target". Apparently scp needs the spaces escaped with a backlash but I don't know how to instruct FIND to print the path that way nor do I know how to insert a SED command in there to do something like: sed 's/ //g'.

The other thing is that when the file to be copied is in a folder that doesn't exist in the target scp also generates an error. So I don't know if I should try with "mkdir -p" before attempting to copy every file (assuiming mkdir -p works on a remote host) or if there is any way to force scp to create the missing folders.

Of course if you can suggest a better solution, by all means!! Thank you.

Upvotes: 0

Views: 963

Answers (2)

Dana Marble
Dana Marble

Reputation: 101

You could try wrapping all of this up in a for loop:

for i in `find /FOLDER -type f -mmin -60`; do scp "$i" user@host:"$i"; done

The example I have here will copy the files you look for to a folder of the same name on the other server. The double quotes around the variable ($i) should help with spaces.

Run this as a cronjob and you might get the result you want. I would be cautious doing so however, you might run into a race condition, by which you would run the second copy procedure before the first one finishes.

If you run the 'time' command before the first run, you can get a sense for how bit a cronjob window to make. Hope this helps.

Upvotes: 0

joschua011
joschua011

Reputation: 4207

This is probably a bit overkill, but if you want to copy files as soon as possible after they had been created or changed take a look at inotify.

inotify can watch directories and notify you when changes occour.

see:http://linux.die.net/man/7/inotify

if you are fit in C i uplaoded a little example for you:

http://pastebin.com/1AUW7fs3

or if you want to stick with bash look at:

inotify and bash

Upvotes: 0

Related Questions