CODEWITHSUNDEEP
pythonrubyperlshellrsync
Neeraj
Neeraj

Reputation: 8532

Parallel rsync on multiple hosts

I have a requirement to fetch a set of files from 15 different hosts. These are in essense just text files.

I have been able to set the ssh keys and stuff so that scp,ssh,rsync etc are passwordless now.

Now I am searching for the right way to fetch these files, i.e fetch them fast and in a manner I can track (i.e not a background process I hope will finish in sometime and then start processing these files)

As of now I am language agnostic. Anything from shell, perl, python, ruby etc would do.

Just though I should ask from the experts here.

Upvotes: 1

Views: 2005

Answers (2)

salva
salva

Reputation: 10234

Try Perl Net::OpenSSH::Parallel:

use Net::OpenSSH::Parallel;

my $pssh = Net::OpenSSH::Parallel->new;
$pssh->add_host($_) for @hosts;

$pssh->push('*', scp_get => '/remote/file/path', '/local/file/path-%HOST%');
$pssh->run;

# do whatever you want with the local files.

Upvotes: 1

charlax
charlax

Reputation: 26049

Use fabric! In particular, you may find the Parallel execution page interesting.

Example:

from fabric.api import env, run

env.user = 'implicit_user'
env.hosts = ['host1', 'explicit_user@host2', 'host3']

def print_user():
    with hide('running'):
        run('echo "%(user)s"' % env)

Output:

$ fab print_user

[host1] out: implicit_user
[explicit_user@host2] out: explicit_user
[host3] out: implicit_user

Done.
Disconnecting from host1... done.
Disconnecting from host2... done.
Disconnecting from host3... done.

Upvotes: 2

Related Questions