rsync - copy files to another server

Question

I have more than 500 Mp4 files in my server 1 so i want half of them to send to server 2 and half of them to server 3

but i dont know how to make this

Is there a way to select files by alphabet or maybe date or something else example videos that start with

 a,c,e*.mp4  

will send to server 2 and videos that start with

 b,d,f*.mp4 

will send to server 3

or is there any other way you think is better

rsync -avzP /home/user/public_html/domain.com/ ip:/home/user2/public_html/domain.com/ 

Answer 1

1) use find to make a list of all the files

find /opt/mymp3folder -print > /tmp/foo

2) find the count of lines and split the list in two

cd /tmp
wc -l /tmp/foo
387
split -l 200 /tmp/foo

3) split by default makes a set of files called xaa xab xac etc. So use xaa to copy to one server and xab to copy to the other

rsync -av --files-from=/tmp/xaa . server1:/opt/newmp3folder/
rsync -av --files-from=/tmp/xab . server2:/opt/newmp3folder/

'.' in the above is the "source" path and allows the use of relative paths in the "files-from" You either need to be in the same path that the find command is run from and use . or set it to an absolute value

Obviously if you wanted to do this on a regular basis probably want to script it properly

Answer 2

I think that is better to split files by size than for numbers (I assume that you have several file sizes in your mp4).

#!/bin/bash
FOLDER=$1
TMP_FILE=$(mktemp)

find  $FOLDER -type f -exec stat -c "%s;%n" {} \; | sort -t ';' -k 2 | awk 'BEGIN{ sum=0; FS=";"} { sum += $1; print sum";"$1";"$2 }' > $TMP_FILE
TOTAL_SIZE=$(tail -n 1 $TMP_FILE | cut -f 1 -d ';')
HALF_SIZE=$(echo $TOTAL_SIZE / 2 | bc)
echo $TOTAL_SIZE $HALF_SIZE


# split part
IFS=';'
while read A B C ; do
    [ $A -lt $HALF_SIZE ] && echo "$C" >> lst_files_1.txt || echo "$C" >> lst_files_2.txt
done  < $TMP_FILE
rsync -avzP 
rm $TMP_FILE

After execution you have list_files_1.txt and list_files_2.txt that contains half of files depending of size.

You can send this files to each server using rsync:

rsync -avzP $(cat list_files_1.txt) ip:/home/user2/public_html/domain.com/

Answer 3

1) use find to make a list of all the files

find /opt/mymp3folder -print > /tmp/foo

2) find the count of lines and split the list in two

wc -l /tmp/foo

387 split -l 200 /tmp/foo

mv xaa xaa.txt

and then rsync like this

 rsync -avzP -e ssh `cat xaa.txt` root@0.0.0.0:/var/www/