CODEWITHSUNDEEP
pythonmultithreadingfuturepython-asyncio
lgvaz
lgvaz

Reputation: 522

Status of future inside loop is always pending

First of all I'm running this on jupyter notebook and that might be the cause of my confusion.

I have a function foo that does some IO operations: 

def foo():
    # Dom some stuff
    time.sleep(1.)
    return 'Finished'

I want to be able to run this function in the background, so I decided to use run_in_executor: 

future = asyncio.get_event_loop().run_in_executor(None, foo)

Now the operation is non blocking and it's all fine, but if this future returns an error I want to catch it and stop the main program, so I decided to write a loop that keeps checking for the status of this future:

while True:
    time.sleep(0.1)
    if future.done():
    # Do some stuff, like raising any caught exception

The problem is that the future status is never being changed, inside the loop it's always pending.

If instead of checking in the loop I manually check the status of the future (in the jupyter notebook) it's going to be correctly marked as finished. This behavior confuses me...

How can I keep checking for the status of a future inside a loop formulation?

Upvotes: 3

Views: 3600

Answers (1)

user4815162342
user4815162342

Reputation: 154846

run_in_executor is designed for use in asyncio, so its return value is an asyncio Future, which is manipulated by the thread that runs the asyncio event loop. Since your code is spinning in a while loop without awaiting anything, it is not giving the event loop a chance to run at all, effectively blocking the event loop. The future remains "pending" because the callback that would update is supposed to be called by the event loop, which is currently not operational - it just sits in a queue.

Replacing time.sleep(0.1) with await asyncio.sleep(0.1) would probably fix the issue. But then you don't need the while loop at all; since as an asyncio future is awaitable, you can await it directly:

await future
# Do some stuff, with the future done.

await suspends the current coroutine until the future is complete, giving other tasks a chance to run in the meantime. It returns the value of the future, or propagates the exception.

The alternative is not to use asyncio at all, but use concurrent.futures directly. That way you get the expected threading semantics (true "background" execution) and a Future that works accordingly.

# keep this in a global variable, so that the same executor
# is reused for multiple calls
executor = concurrent.futures.ThreadPoolExecutor()

# later, submit foo to be executed in the background
future = executor.submit(foo)

This future is a concurrent.futures future which supports waiting for result like this:

result = future.result()

Also, with this kind of future your original code would work unchanged.

Upvotes: 3

Related Questions