CODEWITHSUNDEEP
sshfile-transferscp
Open the way
Open the way

Reputation: 27339

How to efficiently transfer remotely millions of files

In some machine there are almost five millions of small (50KB) text files. I need to send them to another machine on our LAN. I tried doing

scp *.txt remote_machine:

since ssh connection is setup passwordless between them. But a new connection is established for each file, so it is painstakingly slow.

I wonder therefore what would be the best strategy for doing this.

Upvotes: 1

Views: 611

Answers (4)

gucki
gucki

Reputation: 4772

Not sure if scp is multi-threaded. If not try something like this so better utilize all cores/ cpus and network bandwidth:

scp [A-M]*.txt remote_machine:
scp [M-Z]*.txt remote_machine:
scp [0-9]*.txt remote_machine:
...

Of course the patterns to use depend on the naming of your files.

Instead of scp you could also use rsync with the same approach.

Upvotes: 1

dave4420
dave4420

Reputation: 47052

.tar.gz the files together and un.tar.gz the files apart at the other end.

tar cz *.txt | ssh remote_machine 'tar xz'

Ssh itself slows things down. If you are copying between hosts on the same network and security isn't an issue, it may be better to use a raw tcp connection.

remote_machine$ nc -l 3333 -q 1 | tar xz
local_machine$ tar cz *.txt >/dev/tcp/remote_machine/3333

If you want to use a different port number from 3333, make sure you change it in both lines.

Upvotes: 1

P M
P M

Reputation: 857

well ssh also means encyption/decryption, why not you use ftp rather to transfer ... if security is not a real concern ?

more over, you can archive the data and decompress it after transfering ... if your network is slow.

so in short, issue the following command to make archive ..

cd /path/to/transfer/folder
tar -cvpjf /tmp/transfer.tar.bz2 .

to tranfer you will issue command

ftp open remotemachine
put /tmp/transfer.tar.bz2

on reciving, you will issue command in the folder you wanted to issue all...

cd /path/where/to/extract
tar -xvpjf ~/transfer.tar.bz2
rm ~/transfer.tar.bz2

definately you can automate it, i automated this process for me to transfer big chunk of data to a target ...

Upvotes: 1

kev
kev

Reputation: 161684

You can make a files.tar.gz file before transferring.

Upvotes: 4

Related Questions