CODEWITHSUNDEEP
pythonsubprocesspipestdout
vikkyhacks
vikkyhacks

Reputation: 3230

Popen.communicate is stuck until process spawned by sub-process terminates

I have the below three scripts, and when I run main.py file, it spawns child.py which again executes the subchild.py and terminates quicly, subchild.py however keeps executing for a lot of time.

The problem with this is, main.py is blocked at p.communicate() till subchild.py terminates. If I open task manager and kill the running subchild.py main.py immediately returns the output of child.py

So my questions are as below

    # main.py file

    if __name__ == '__main__':
        import subprocess
        p = subprocess.Popen(
            ["python", "child.py"],
            stdout=subprocess.PIPE,
            stdin=subprocess.PIPE,
            stderr=subprocess.PIPE
        )
        out, _ = p.communicate()
        print(out)


    # child.py file

    if __name__ == '__main__':
        import subprocess
        print("Child running")
        p = subprocess.Popen(["python", "subchild.py"])
        print("Child pid - ", p.pid)
        exit(1)


    # subchild.py file

    if __name__ == '__main__':
        import time
        time.sleep(10000)

Note: I'm trying this on Windows 7 Enterprise. I'm using python3.6.6

Update after comment: On the main.py I need child.py's pid, stdout, stderr and process object so that I can kill child.py from main.py at any later point if I want to. This code is a small snippet is part of some API which I am building where the user would want the control to kill the process if he wishes to. subprocesses.call or subprocesses.run would not let me get control over the process object. I also won't have control over what child.py command I will receive as input for main.py, So I need to somehow not wait for the subchild.py and exit immediately with child.py's output as soon as it completes.

Upvotes: 3

Views: 1299

Answers (3)

vikkyhacks
vikkyhacks

Reputation: 3230

Finally I found the solution to this myself, this seems to be an open issue with python. Would be glad to see this fixed. https://bugs.python.org/issue26731

Upvotes: 1

LeKhan9
LeKhan9

Reputation: 1350

Right now, child.py will always wait for subchild.py to finish. If you don't need to wait, you can try subprocess.call() as an alternative in the child.py code. Replace the Popen() logic with the call() logic instead. 

p = subprocess.call(["python", "subchild.py"])

Here is a reference: https://docs.python.org/2/library/subprocess.html#subprocess.call

An alternative is to keep the code as it is an actually change to using .call() in the main.py file. 

import subprocess
p = subprocess.call(
    ["python", "child.py"],
    stdout=subprocess.PIPE,
    stdin=subprocess.PIPE,
    stderr=subprocess.PIPE
)

You mentioned you can't use .call() because you'll need to kill it, however this is not the case. Making this change should allow you to:

  • return main.py and child.py quickly,
  • it will print the ID of the subchild process from child.py and
  • subchild.py will continue running

Upvotes: 1

user2357112
user2357112

Reputation: 280181

Your communicate call doesn't just wait for the child to complete. It first tries to read the entire contents of the child's stdout and stderr.

Even when the child has completed, the parent can't stop reading, because the grandchild inherits the child's stdin, stdout, and stderr. The parent needs to wait for the grandchild to complete, or at least for the grandchild to close its stdout and stderr, before the parent can be sure it's done reading.

Upvotes: 2

Related Questions