CODEWITHSUNDEEP
pythonssh
Cyberwiz
Cyberwiz

Reputation: 11426

Output garbled when launching multiple ssh-sessions with pseudo-tty (need remote process to exit when ssh disconnects/is killed)

I have a python script that opens multiple concurrent pseudo-tty ssh sessions to a server. My problem is that the output is garbled:

for i in range(0, 3):
    subprocess.Popen(
        "ssh -tt -q myserver 'echo 11; echo 22; echo 33; echo 44;'",
        shell=True
    )

Output:

    11
      22
        33
          44
            11
    22
    33
    44
    11
    22
    33
    44

The output varies. Sometimes it works, but most of the time I get those weird indentations. In reality I want to launch remote python processes (a locust load gen slave), but I've simplified it to just use echo.

Things I've tried:

11^M$
     22^M$
          33^M$
               44^M$
                    11$
22$
33$
44$
11$
   22$
      33$
         44$

I'm not sure if is even a python issue or just an SSH issue. My guess is that I need to use some sort of line buffering, but I dont know how :-/

I'm on MacOS Mojave, and I've tried both in iTerm2 and Term if that matters.

Edit: I'm not sure it is related, but the problem appears to occur more frequently if I ensure python keeps running until the ssh session has terminated (by adding time.sleep(10) at the end of the script)

edit 2: I tried @FLemaitre 's solution (not using -tt and killing explicitly), and it works in the simple case, but not when spawning locust:

proc = subprocess.Popen(
    "ssh servername 'locust --slave --master-port 7777 --no-web -f locustfile.py & read; kill $!'",
    shell=True,
    stdin=subprocess.PIPE,
)
time.sleep(10)
proc.kill()
proc.wait()

On the remote a bash -c locust --slave ... process is started. It dies when ssh is killed, but locust itself (a child of the above process) does not :-/

Upvotes: 4

Views: 333

Answers (1)

Francois Lemaitre
Francois Lemaitre

Reputation: 511

I reproduce systematically the issue with the following script:

import subprocess
import time

if __name__ == "__main__":
    for i in range(0, 10):
        proc = subprocess.Popen(
            "ssh -tt -q localhost 'echo 11; echo 22; echo 33; '",
            shell=True
        )
    time.sleep(4)

And I think the issue is not related to Python. These multiple ssh with pseudo-TTY seem to conflict with each other's. Eventually, the terminal used to run this script ends up broken as well (whereas it wasn't sourced):

>cat test2.py
import subprocess
                 import time
                            import atexit
... etc ...

I checked the documentation and this -t option seems to do much more than what you are actually trying to achieve. When I remove the second t and the -q options, I sometimes (not often), get a cryptic error message stating that something went wrong (but I no longer manage to reproduce it). I checked with google but without much success. Still, I'm convinced that this option is overkill and I would rather focus on the undying processes. This one issue is well known:

Starting a process over ssh using bash and then killing it on sigint

The second answer is your -tt option, but the best answer suits your example very well and is superior (with -tt you solve the ssh propagation of the termination but do not tackle the same issue between Python and its subprocess). For example:

import subprocess
import time

if __name__ == "__main__":
    for i in range(0, 10):
        proc = subprocess.Popen(
            "ssh localhost 'sleep 90 & read ; kill $!'",
            shell=True,
            stdin=subprocess.PIPE
        )
    time.sleep(40)

With this solution, stdin is shared by all actors (python, the python subprocess, the ssh process, the sleep process), and its closure at any point in the chain is detected by the final business process, trigering a graceful shutdown.

Edit with locust: I gave it a quick try and the issue was that a simple 'kill' is ignored by the slave (looks like an issue on lucust side). It seems to work with a 'kill -9':

import subprocess
import time

if __name__ == "__main__":
    for i in range(0, 2):
        proc = subprocess.Popen(
            "ssh localhost 'python -m locust --slave --no-web -f ~devsup/users/flemaitre/tmp/locust_config.py & read ; kill -9 $!'",
            shell=True,
            stdin=subprocess.PIPE
        )
    time.sleep(40)

Upvotes: 2

Related Questions