CODEWITHSUNDEEP
pythonsubprocessmultiprocessingfabric
loganasherjones
loganasherjones

Reputation: 1060

Running Fabric command in Seperate Process Group Hanging

I'm having trouble understanding exactly why this is hanging. I have stripped down this example to the core components. I have a file, let's call it do_ls.py

import fabric.api
import time

host = "myhost.mydomain"
username = "username"
password = "password"

def main():
    with fabric.api.settings(host_string=host,user=username,password=password):
        result = fabric.api.run("ls")

if __name__ == "__main__":
    main()

If I run this command: python do_ls.py it will execute correctly. Now for the problem. I would like to run this in it's own process. So I have this file, let's call it main.py

import sys
import os
import logging
import subprocess as sp
import time

def main():
    logging.basicConfig(level=logging.DEBUG)
    cmd = [sys.executable, "/path/to/do_ls.py"]
    p = sp.Popen(cmd, preexec_fn=os.setpgrp)
    while p.poll() is None:
        print "Sleeping..."
        time.sleep(0.5)
    print "All Done."

if __name__ == "__main__":
     main()

Now if I run python main.py this will hang forever. The problem as far as I know is that I'm running the process in a subgroup (i.e. if I take out preexec_fn=os.setpgrp then it will work correctly). What I don't understand is, why this is the case. Especially given that the following works:

    cmd = ["ssh", "-q", "username@hostname", "ls"]
    p = sp.Popen(cmd, preexec_fn=os.setpgrp)

Any insight would be greatly appreciated.

Upvotes: 2

Views: 202

Answers (1)

Ryan Schuster
Ryan Schuster

Reputation: 506

Since the following lines work:

cmd = ["ssh", "-q", "username@hostname", "ls"]
p = sp.Popen(cmd, preexec_fn=os.setpgrp)

but main.py continuously hangs, I assume that 

while p.poll() is None:

never evaluates to False. So p.poll() must always be returning None, possibly even after the process completes. A quick search returned this conversation on Python's bug reporting site. As per that conversation, try calling sp.Popen() with the (undocumented) _deadstate='dead' option:

The problem is that os.wait() does not play nicely with subprocess.py. Popen.poll() and Popen.wait() use os.waitpid(pid, ...) which will raise OSError if pid has already been reported by os.wait(). Popen.poll() swallows OSError and by default returns None.

You can (sort of) fix your program by using p.popen(_deadstate='dead') in place of p.popen(). This will make poll() return 'dead' instead of None if OSError gets caught, but this is undocumented.

Upvotes: 1

Related Questions