CODEWITHSUNDEEP
python-3.xstdoutpython-asyncio
user3535074
user3535074

Reputation: 1338

Asyncio stdout - failing

I'm trying to get to grips with using asyncio to speed up a process of calling an external tool to analyse multiple audio files. I'm working on windows, python 3.6 (anaconda installation) and the problem I'm having here is that the calls don't seem to wait or the results are never received through stdout.

Any ideas?

import os
import asyncio

external_tool = r"C:\path\to\external\tool.exe"

def filepaths_generator(root):
    '''
    Simple generator to yield filepaths
    '''
    for path, dirs, files in os.walk(root):
        for f in files:
            yield os.path.join(path,f)


async def async_subprocess_command(*args):
    '''
    A function to run the asyncio subprocess call
    '''
    # Create subprocess
    process = asyncio.create_subprocess_exec(
        *args,
        # stdout must a pipe to be accessible as process.stdout
        stdout=asyncio.subprocess.PIPE)
    # Wait for the subprocess to finish
    stdout, stderr = await process.communicate()
    # Return stdout
    return stdout.decode().strip()

async def add_external_tool_data_to_dict(filepath):
    external_tool_data = await async_subprocess_command(external_tool,filepath)
    metadata_dict[filepath] = external_tool_data
    return

metadata_dict = {}

loop = asyncio.get_event_loop()
#loop = asyncio.ProactorEventLoop() - Tried this, it doesn't help!

filepaths = filepaths_generator(r"C:\root\path")
tasks = []

for filepath in filepaths:
    #Create tasks in for the loop and add to a list
    task = loop.create_task(add_external_tool_data_to_dict(filepath))
    tasks.append(task)
#Run the tasks
loop.run_until_complete(asyncio.wait(tasks))

loop.close()
print(metadata_dict)

Upvotes: 2

Views: 1044

Answers (2)

user3535074
user3535074

Reputation: 1338

As per Gerrat's comment, the solution was to use the ProactorEventLoop and add an additional line thereafter:

loop = asyncio.ProactorEventLoop()
asyncio.set_event_loop(loop)

Upvotes: 1

Gerrat
Gerrat

Reputation: 29730

For starters there's this:

On Windows, the default event loop is SelectorEventLoop which does not support subprocesses. ProactorEventLoop should be used instead

Example to use it on Windows:

import asyncio, sys

if sys.platform == 'win32':
    loop = asyncio.ProactorEventLoop()
    asyncio.set_event_loop(loop)

I would start by switching to this alternate event loop.

Upvotes: 6

Related Questions