CODEWITHSUNDEEP
pythonpython-3.xsubprocess
ananvodo
ananvodo

Reputation: 381

Why does my python script get frequently terminated when using subprocess?

I have this code. Basically I using subprocess to execute a program several times in a while loop. It works fine but after several times (5 times to be precise) the my python script just terminates and it still has a long way before finishing.

        while x < 50:

            # ///////////I am doing things here/////////////////////

            cmdline = 'gmx mdrun -ntomp 1 -v -deffnm my_sim'
            args = shlex.split(cmdline)
            proc = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
            output = proc.communicate()[0].decode()

            # ///////////I am doing things here/////////////////////
            x += 1

For each time I am calling program, it will take approximately one hour to finish. In the mean time subprocess should wait because depending on the output I must execute parts of my code (that is why I am using .communicate() ).

Why is this happening?

Thanks for the help in advanced!

Upvotes: 1

Views: 306

Answers (2)

ananvodo
ananvodo

Reputation: 381

To solve this problem I suggest doing the following:

    while x < 50:

        # ///////////I am doing things here/////////////////////

        cmdline = 'gmx mdrun -ntomp 1 -v -deffnm my_sim 2>&1 | tee output.txt'
        proc = subprocess.check_output(args, shell=True)
        with open('output.txt', 'r') as fin:
        out_file = fin.read()

        # ///////////Do what ever you need with the out_file/////////////   


        # ///////////I am doing things here/////////////////////
        x += 1

I know it is not recommended to use shell=True, so if you do not want to use it then just pass the cmdline you with commas. Be aware that you might get an error when passing with commas. I do want to go into details but in that case you can use shell=True and your problem will be gone.

Using the piece of code I just provided your python script will not terminate abruptly when using subprocess many time and with programs that have a lot of stdout and stderr messages.

It tool some time to discover this and I hope I can help someone out there.

Upvotes: 0

Kristopher Ives
Kristopher Ives

Reputation: 6045

A subprocess runs asynchronously in the background (since it's a different process) and you need to use subprocess.wait() to wait for it to finish. Since you have multiple subprocesses you'll likely want to wait on all of them, like this:

exit_codes = [p.wait() for p in (p1, p2)]

Upvotes: 1

Related Questions