CODEWITHSUNDEEP
pythonsubprocess
SomethingSomething
SomethingSomething

Reputation: 12178

Python: Popen - wait for main process, but not for background subprocesses

I work in Unix, and I have a "general tool" that loads another process (GUI utility) on the background, and exits.

I call my "general tool" from a Python script, using Popen and proc.communicate() method.

My "general tool" runs for ~1 second, loads the GUI process on the background and exits immediately.

The problem is that proc.communicate() continues waiting to the process, although it's already terminated. I have to manually close the GUI (which is a subprocess that runs on the BG), so proc.communicate() returns.

How can this be solved?

I need proc.communicate() to return once the main process is terminated, and not to wait for the subprocesses that run on the background...

Thanks!!!

EDIT:

Adding some code snippets:

My "General Tool" last Main lines (Written in Perl):

if ($args->{"gui"}) {
    my $script_abs_path = abs_path($0);
    my $script_dir = dirname($script_abs_path);
    my $gui_util_path = $script_dir . "/bgutil";
    system("$gui_util_path $args->{'work_area'} &");
}
return 0;

My Python script that runs the "General Tool":

cmd = PATH_TO_MY_GENERAL_TOOL
proc = subprocess.Popen(cmd, shell = True, stdout = subprocess.PIPE, stderr = subprocess.STDOUT)
stdout, dummy = proc.communicate()
exit_code = proc.returncode
if exit_code != 0:
    print 'The tool has failed with status: {0}. Error message is:\n{1}'.format(exit_code, stdout)
    sys.exit(1)

print 'This line is printed only when the GUI process is terminated...'

Upvotes: 1

Views: 1809

Answers (1)

Dunes
Dunes

Reputation: 40693

Don't use communicate. Communicate is explicitly designed to wait until the stdout of the process has closed. Presumably perl is not closing stdout as it's leaving it open for it's own subprocess to write to.

You also don't really need to use Popen as you're not really using its features. That is, you create pipes, and then just reprint to stdout with your own message. And it doesn't look like you need a shell at all.

Try using subprocess.call or even subprocess.check_call.

eg.

subprocess.check_call(cmd)

No need to check the return value as check_call throws an exception (which contains the exit code) if the process returns with a non-zero exit code. The output of the process is directly written to the controlling terminal -- no need to redirect the output.

Finally, if cmd is a compound of a path to an executable and its arguments then use shlex.split.

eg.

cmd = "echo whoop" # or cmd = "ls 'does not exist'"
subprocess.check_call(shlex.split(cmd))

Sample code to test with:

mypython.py

import subprocess, shlex
subprocess.check_call(shlex.split("perl myperl.pl"))
print("finishing top level process")

myperl.pl

print "starting perl subprocess\n";
my $cmd = 'python -c "
import time
print(\'starting python subprocess\')
time.sleep(3);
print(\'finishing python subprocess\')
" &';
system($cmd);
print "finishing perl subprocess\n";

Output is:

$ python mypython.py 
starting perl subprocess
finishing perl subprocess
finishing top level process
$ starting python subprocess
finishing python subprocess

Upvotes: 2

Related Questions