Parallel copying

Question

I need a program, which can do parallel copying to many servers. My data is about 70 gigabytes and i need copy that to many another servers. If i do rsync to one server, the time is about 90 minutes, rsync to 2 servers longs about 120 minutes, and rsync to 3 servers can takes 200 minutes. But each of rsync processes read directory, so the speed of reading is slow, when there are many rsyncs. Is there any program on Linux, which can do parallel copying - it should read files in directory one time and copy that files to many servers simultaneously. Or may be someone did that on python, for example? Or something as tee command?

Answer 1

The reason why it takes longer when you do multiple transfers is that the network is the bottle neck, and so transmitting the data to multiple machines causes contention on the senders network interface.

Your problem can be solved by using a multicast transfer protocol. Multicast allows one to send data to multiple receivers with a single transmission, which removes the contention on the senders network interface.

Options include uftp, mrsync, and udpcast