CODEWITHSUNDEEP
pythonsshparallel-processing
Vic Podestà
Vic Podestà

Reputation:

Parallel SSH in Python

I wonder what is the best way to handle parallel SSH connections in python. I need to open several SSH connections to keep in background and to feed commands in interactive or timed batch way. Is this possible to do it with the paramiko libraries? It would be nice not to spawn a different SSH process for each connection.

Thanks.

Upvotes: 3

Views: 7824

Answers (6)

jldugger
jldugger

Reputation: 2469

I've tried clusterssh, and I don't like the multiwindow model. Too confusing in the common case when everything works.

I've tried pssh, and it has a few problems with quotation escaping and password prompting.

The best I've used is dsh:

 Description: dancer's shell, or distributed shell
 Executes specified command on a group of computers using remote shell
 methods such as rsh or ssh.
 .
 dsh can parallelise job submission using several algorithms, such as using
 fan-out method or opening as much connections as possible, or
 using a window of connections at one time.
 It also supports "interactive mode" for interactive maintenance of
 remote hosts.
 .
 This tool is handy for administration of PC clusters, and multiple hosts.

Its very flexible in scheduling and topology: you can request something close to a calling tree if need be. But the default is a simple topology of one command node to many leaf nodes.

http://www.netfort.gr.jp/~dancer/software/dsh.html

Upvotes: 1

Jasim
Jasim

Reputation: 480

This might not be relevant to your question. But there are tools like pssh, clusterssh etc. that can parallely spawn connections. You can couple Expect with pssh to control them too.

Upvotes: -1

Mr_Pink
Mr_Pink

Reputation: 109347

Yes, you can do this with paramiko.

If you're connecting to one server, you can run multiple channels through a single connection. If you're connecting to multiple servers, you can start multiple connections in separate threads. No need to manage multiple processes, although you could substitute the multiprocessing module for the threading module and have the same effect.

I haven't looked into twisted conch in a while, but it looks like it getting updates again, which is nice. I couldn't give you a good feature comparison between the two, but I find paramiko is easier to get going. It takes a little more effort to get into twisted, but it could be well worth it if you're doing other network programming.

Upvotes: 3

ars
ars

Reputation: 123498

It might be worth checking out what options are available in Twisted. For example, the Twisted.Conch page reports:

http://twistedmatrix.com/users/z3p/files/conch-talk.html

Unlike OpenSSH, the Conch server does not fork a process for each incoming connection. Instead, it uses the Twisted reactor to multiplex the connections.

Upvotes: 3

ewanm89
ewanm89

Reputation: 929

Reading the paramiko API docs, it looks like it is possible to open one ssh connection, and multiplex as many ssh tunnels on top of that as are wished. Common ssh clients (openssh) often do things like this automatically behind the scene if there is already a connection open.

Upvotes: 1

vog
vog

Reputation: 25597

You can simply use subprocess.Popen for that purpose, without any problems.

However, you might want to simply install cronjobs on the remote machines. :-)

Upvotes: 1

Related Questions