CODEWITHSUNDEEP
pythonpython-asynciosemaphore
John
John

Reputation: 185

asyncio.Semaphore(2) does not limit amount of subprocesses to 2

I am using asyncio throughout my application. In the application an external program is started, but I want to limit the maximum amount of external concurrent processes to 2, that's why I am using Semaphore(2) like so:

async def myfunc(args):
    proc = await asyncio.create_subprocess_shell(args)
    returncode = await proc.wait()

sem = asyncio.Semaphore(2)
async with sem:
    result = await myfunc("some arguments")

When I send 3 requests one after the other, and I check top command (it's on Linux), I see 3 processes running at the same time. I expected to see 2 process running at the same time, and then as soon as one of those is done, the third one will be started. Am I using Semaphore(2) incorrectly?

Upvotes: 0

Views: 746

Answers (1)

Menglong Li
Menglong Li

Reputation: 2255

You should write codes like this, make sure the semaphore uses the same loop with your myfunc, the following codes works fine with me:

import asyncio


async def myfunc(cmd, sem):
    async with sem:
        print('start')
        proc = await asyncio.create_subprocess_shell(cmd)
        returncode = await proc.wait()
        return returncode


async def main():
    sem = asyncio.Semaphore(2)
    cmd = 'python -c "import time; time.sleep(3)"'
    tasks = [myfunc(cmd, sem), myfunc(cmd, sem), myfunc(cmd, sem)]
    results = await asyncio.gather(*tasks, return_exceptions=True)
    assert results == [0, 0, 0]


if __name__ == '__main__':
    asyncio.run(main())

Upvotes: 1

Related Questions