Andy W
Andy W

Reputation: 11

Linux parallel copy from one source to multi destinations with a progress output

I'm trying to parallel copy one large dataset (~7TB) to three external USB drives at the same time. Ideally I want to be able to kick it off, leave it to crack on and come back to check on it every so often.

I understand that it may not be the quickest way to do it, but I really don't want to have to go back to swap drives over and take 2 days to do it.

Here's where I've got to:

parallel --bar rsync ./source/ -r --info=progress2 ::: /dest/one/ /dest/two/

Which works, but I cannot for the life of me figure out how to get a good readout. There are plenty of parallel copying of one dataset for speed, but my limitation is the USB HDD write speed, which is about 200MBps.

I'm absolutely open to suggestions that throw the current idea in the bin.

EDIT

The command I ended up with thanks to Renaud is:

parallel -u rsync ./source/ -r --info=progress2{2} {1} ::: /dest/one/ /dest/two/

Which gives me a progress report of one of the threads which isn't perfect for all applications, but is great for this one!

Upvotes: 1

Views: 243

Answers (1)

Ole Tange
Ole Tange

Reputation: 33685

Which works, but I cannot for the life of me figure out how to get a good readout.

Is progress information of each rsync what you are asking for?

In that case try:

parallel --ll --tag rsync ./source/ -r --info=progress2 ::: /dest/one/ /dest/two/

Requires version 20220522 or later.

Upvotes: 0

Related Questions