CODEWITHSUNDEEP
linuxbashshellunixrsync
Flex Brannigan
Flex Brannigan

Reputation: 31

Rsync using find command, delete original, replace original with symlink

I work for a graphics company, and our clients submit massive amounts of data. A single project folder can take up hundreds of gigabytes of space on our server. While space on my direct attached RAID is limited to 10TB, I have a 96TB storage server connected to my main server via iSCSI.

I'm not versed enough in BASH scripting to automate this process, but what I do manually is perform a find with this command:

find /disk -type d -name 09\ Client-Files | sed 's/ /\\ /g' > client-origs.txt

From this list I rsync the "09 Client-Files" directory to the storage server using:

rsync -avR /disk/Client-Jobs/1234\ Client/09\ Client-Files /iscsi

With the -R option it's copying the relative path to /iscsi.

After this the original "09 Client-Files" directory is deleted and a symlink pointing to its new location on the /iscsi volume.

What I'm shooting for is to use the found path as a variable to do everything at once. I'm really at a loss on how to proceed. I've been trying to educate myself on bash variables for days now — I can get them to work on simpler things but in this case I'm coming up empty. Can you help a brother out?

Upvotes: 3

Views: 3158

Answers (2)

RedComet
RedComet

Reputation: 1202

You can do all that at once with find:

find /usr -type d -name My\ Files -exec rsync -avR '{}' /iscsi \;

For more info man find, right before the end there are several examples. Alternatively you can pipe the results using the xargs utility.

You should explore your manuals more througly, there is a lot of treasure to be found.

On a bash script you could also put:

export MIVARIABLE=$(find /usr -type d -name My\ files)

and then 

for i in $MIVARIABLE ;do
   rsync -avR $i /iscsi;
done

But this has drawbacks among them if your results are very big, it may exceed bash maximum text space available for a variable , and if the retrieved names include spaces, some utilities may go crazy.

In a modern system the maximum size on a bash environment variable can be from 8kib to nearly 2MiB, if you put to much info on a variable you'll get a Argument list too long error.

So if you need to iterate over a file/directory list you better use find built-in functions or in cooperation with xargs. Read carefully their manuals and try some dry runs using echo your_command_line instead, so you gain appropiate experience and confidence with the utilities. Look for the -print0 function on find as well as the -0 command on xargs which are designed to avoid some problems with filenames with spaces and symbols in their names.

EDIT: find will handle white spaces correctly with the -exec operator:

find /usr -type d -name My\ Files -exec rsync -avR '{}' /iscsi \;  -exec rm -rf '{}'\;

would do what you need one after the other. I was pointing out options, so you could decide what was better suited to your needs, I did not meant to confuse you.

Edit2:

find /usr -type d -name 'My Files' -exec rsync -avR '{}' /iscsi \;  -exec rm -rf '{}'\;

funciona igual que el ejemplo de arriba. Cuando bash lee 'My Files' it takes the whole word togheter and does not treat the white spaces inside differently

Upvotes: 0

user783774
user783774

Reputation:

have you looked at the lndir command? (may need to be installed separately)

It creates a shadow directory of symbolic links pointing to another directory tree.

Upvotes: 1

Related Questions