CODEWITHSUNDEEP
linuxscp
Philip
Philip

Reputation: 395

How to copy multiple files simultaneously using scp

I would like to copy multiple files simultaneously to speed up my process I currently used the follow

scp -r [email protected]:/var/www/example/example.example.com .

but it only copies one file at a time. I have a 100 Mbps fibre so I have the bandwidth available to really copy a lot at the same time, please help.

Upvotes: 8

Views: 65484

Answers (6)

Anuruddha Lanka Liyanarachchi
Anuruddha Lanka Liyanarachchi

Reputation: 2209

You can use background task with wait command. Wait command ensures that all the background tasks are completed before processing next line. i.e echo will be executed after scp for all three nodes are completed.

#!/bin/bash
scp -i anuruddha.pem myfile1.tar [email protected]:/tmp &
scp -i anuruddha.pem myfile2.tar [email protected]:/tmp &
scp -i anuruddha.pem myfile.tar [email protected]:/tmp &

wait
echo "SCP completed"

Upvotes: 9

BhEaN
BhEaN

Reputation: 91

You can use parallel-scp (AKA pscp): http://manpages.ubuntu.com/manpages/natty/man1/parallel-scp.1.html

With this tool, you can copy a file (or files) to multiple hosts simultaneously.

Regards,

Upvotes: 3

kranthi117
kranthi117

Reputation: 628

I am not sure if this helps you, but I generally archive (compression is not required. just archiving is sufficient) file at the source, download it, extract them. This will speed up the process significantly. Before archiving it took > 8 hours to download 1GB After archiving it took < 8 minutes to do the same

Upvotes: 4

Mike S.
Mike S.

Reputation: 2111

SSH is able to do so-called "multiplexing" - more connections over one (to one server). It can be one way to afford what you want. Look up keywords like "ControlMaster"

Second way is using more connections, then send every job at background:

for file in file1 file2 file3 ; do 
     scp $file server:/tmp/ & 
done

But, this is answer to your question - "How to copy multiple files simultaneously". For speed up, you can use weaker encryption (rc4 etc) and also don't forget, that the bottleneck can be your hard drive - because SCP don't implicitly limit transfer speed.

Last thing is using rsync - in some cases, it can be lot faster than scp...

Upvotes: 6

GNSL
GNSL

Reputation: 111

If you specify multiple files scp will download them sequentially: 

scp -r [email protected]:/var/www/example/file1 [email protected]:/var/www/example/file2 .

Alternatively, if you want the files to be downloaded in parallel, then use multiple invocations of scp, putting each in the background.

#! /usr/bin/env bash
scp [email protected]:/var/www/example/file1 . &
scp [email protected]:/var/www/example/file2 . &

Upvotes: 0

Aaron Digulla
Aaron Digulla

Reputation: 328594

100mbit Ethernet is pretty slow, actually. You can expect 8 MiB/s in theory. In practice, you usually get between 4-6 MiB/s at best.

That said, you won't see a speed increase if you run multiple sessions in parallel. You can try it yourself, simply run two parallel SCP sessions copying two large files. My guess is that you won't see a noticeable speedup. The reasons for this are:

  1. The slowest component on the network path between the two computers determines the max. speed.
  2. Other people might be accessing example.com at the same time, reducing the bandwidth that it can give you
  3. 100mbit Ethernet requires pretty big gaps between two consecutive network packets. GBit Ethernet is much better in this regard.

Solutions:

  1. Compress the data before sending it over the wire
  2. Use a tool like rsync (which uses SSH under the hood) to copy on the files which have changed since the last time you ran the command.
  3. Creating a lot of small files takes a lot of time. Try to create an archive of all the files on the remote side and send that as a single archive.

The last suggestion can be done like this:

ssh root@xxx "cd /var/www/example ; tar cf - example.example.com" > example.com.tar

or with compression:

ssh root@xxx "cd /var/www/example ; tar czf - example.example.com" > example.com.tar.gz

Note: bzip2 compresses better but slower. That's why I use gzip (z) for tasks like this.

Upvotes: 4

Related Questions