CODEWITHSUNDEEP
python-3.xpython-asyncio
bottlecap
bottlecap

Reputation: 575

How do I properly use asyncio.create_subprocess_shell() in an async def function?

I'm trying to write a simple subprocess program that will call a long-running shell command, allow other processes to run, and then execute a few clean-up tasks once it is complete.

Unfortunately, I'm running into errors just getting the shell command to execute properly within the asyncio event loop. The behavior is that it looks like it python never waits for the shell script to complete running. I know the shell script works as I can run it manually from the prompt. The shell script I'm running should execute in about 3-5 minutes.

Here's my sample program:

import asyncio 
from asyncio.subprocess import PIPE, STDOUT 
import subprocess 
import signal


def signal_handler(signal, frame):
    loop.stop()
    client.close()
    sys.exit(0)

async def run_async(loop = ''):
    cmd = 'sudo long_running_cmd --opt1=AAAA --opt2=BBBB'

    print ("[INFO] Starting script...")
    await asyncio.create_subprocess_shell(cmd1, stdin = PIPE, stdout = PIPE, stderr = STDOUT)
    print("[INFO] Script is complete.")


loop = asyncio.get_event_loop() 
signal.signal(signal.SIGINT, signal_handler) 
tasks = [loop.create_task(run_async())] 
wait_tasks = asyncio.wait(tasks) 
loop.run_until_complete(wait_tasks)

loop.close()

The program runs and fails almost instantaneously. The error that this code generates is:

[INFO] Starting script...
[INFO] Script is complete.
Exception ignored in: <bound method BaseSubprocessTransport.__del__ of <_UnixSubprocessTransport closed pid=5652 running stdin=<_UnixWritePipeTransport closing fd=7 open> stdout=<_UnixReadPipeTransport fd=8 open>>>
Traceback (most recent call last):
  File "/usr/lib/python3.5/asyncio/base_subprocess.py", line 126, in __del__
  File "/usr/lib/python3.5/asyncio/base_subprocess.py", line 101, in close
  File "/usr/lib/python3.5/asyncio/unix_events.py", line 568, in close
  File "/usr/lib/python3.5/asyncio/unix_events.py", line 560, in write_eof
  File "/usr/lib/python3.5/asyncio/base_events.py", line 497, in call_soon
  File "/usr/lib/python3.5/asyncio/base_events.py", line 506, in _call_soon
  File "/usr/lib/python3.5/asyncio/base_events.py", line 334, in _check_closed
RuntimeError: Event loop is closed

I'm running python v3.5.2 on Ubuntu 16.04.

UPDATE

Based upon Sam's comment below, I needed to update my code to the following in order for it to work:

process = await asyncio.create_subprocess_shell(cmd1, stdin = PIPE, stdout PIPE, stderr = STDOUT)
await process.wait()

This is a slight modification from his code, but works.

Upvotes: 8

Views: 15038

Answers (1)

Sam Hartman
Sam Hartman

Reputation: 6509

The issue is that nothing waits for the process to complete; you only wait for it to start.

async def run_async(loop = ''):
    cmd = 'sudo long_running_cmd --opt1=AAAA --opt2=BBBB'

    print ("[INFO] Starting script...")
    process = await asyncio.create_subprocess_shell(cmd1, stdin = PIPE, stdout = PIPE, stderr = STDOUT)
    await process.wait()
    print("[INFO] Script is complete.")

Upvotes: 10

Related Questions