CODEWITHSUNDEEP
filesynchronizationcopyrsyncscp
Juan Carlos Coto
Juan Carlos Coto

Reputation: 12564

What is a good pattern to synchronize files between computers in parallel (in CentOS)?

Trying to find a good way to copy code between one "deployment" computer and several "target" computers, hopefully in parallel. The idea is that the deployment computer holds a copy of the files as they are supposed to be copied to the target servers. We would like to have copying happen in parallel, as it might involve several tens of target servers.

Our current scheme involves using rsync to synchronize the containing directory where the files reside, in order to keep the target servers up-to-date on the deployment server.

So, the questions are:

  1. What is a good / better way to do this?
  2. What sort of tools are used to do this?
  3. Should this problem be faced from a different angle or perspective that I'm totally missing?

Thanks very much!

Upvotes: 2

Views: 1037

Answers (3)

rread
rread

Reputation: 159

Another option is pdsh, a parallel, distributed shell. It's available from EPEL, and allows running remote commands (via ssh) on multiple nodes in parallel. For example:

pdsh -w node10,node11,node12 command

Runs "command" on all three nodes in parallel. It also has a handy hostname expression feature to do the same thing with a bit less typing:

pdsh -w node[10-12] command

It also includes the pdcp command copies files to multiple nodes in parallel. (The pdsh package needs to be installed on all nodes for pdcp to work.)

pdcp -w node[10-12] /local/file /remote/dir/

The local file is copied to the /remote/dir on all three nodes.

Upvotes: 3

Eric
Eric

Reputation: 251

There is a program called clusterssh that is available on debian based operating systems (but I was able to install it onto RHEL 6.3 using an RPM and resolving other dependencies) that will allow you to open an ssh terminal for multiple machines, with a single input location (this allows you type once onto as many machines as you have terminals open). Then you just have to use a simple scp. I have used this program to move a file from a development workstation to as many as 25 other workstations at the same time, but this option is only really useful if you're trying to accomplish what you stated in the question, that is, copying files from one computer to several others.

This is not an effective syncing mechanism. If you really want it to sync then the above answer would be best.

Upvotes: 1

Thought Space Designs
Thought Space Designs

Reputation: 429

We use the lftp command to sync our remote web server to our local backup machine. We wrote a BaSH script to automatically sync all backups on the server to the local box, and we set that script up on a cron to run nightly.

rsync is a fine way of handling this, and I might recommend moving your current protocol into a cron setup if it isn't already.

Unison is also a tool available for setting up two way sync, if you requie that functionality.

Hope this helps!

Upvotes: 2

Related Questions