CODEWITHSUNDEEP
python-3.xpython-multithreadingpython-asyncio
epeleg
epeleg

Reputation: 10945

using asyncio and threads

Would it make sense to use both asyncio and threading in the same python project so that code runs in different threads where is some of them asyncio is used to get a sequentially looking code for asynchronous activities?

or would trying to do this mean that I am missing some basic concept on the usage of either threading or asyncio?

Upvotes: 2

Views: 6911

Answers (2)

Ytsen de Boer
Ytsen de Boer

Reputation: 3097

Sure it may make sense.

Asynchronous code in principle runs a bunch of routines in the same thread.

This means that the moment one routine has to wait for input or output (I/O) it will halt that routine temporarily and simply starts processing another routine until it encounters a wait there, etc.

Multi-threaded (or "parallelized" code) runs in principle at the same time on different cores of your machine. (Note that in Python parallel processing is achieved by using multiple processes as pointed out by @Yassine Faris below).

It may make perfect sense to use both in the same program. Use asyncio in order to keep processing while waiting for I/O. Use multi-threading (multi processing in Python) to do, for example, heavy calculations in parallel in another part of your program.

Upvotes: 3

Mikhail Gerasimov
Mikhail Gerasimov

Reputation: 39606

I didn't understand what you're asking (part about "sequentially looking code for asynchronous activities"), but since there's no answers I'll write some thoughts.

Let's talk why we need asyncio/threads at all. Imagine we have a task to make two requests.

  1. If we will use plain one-thread non-async code, only option for us is to make request for one url and only after it's done - for another:

    request(url1)
request(url2)

    Problem here is that we do job ineffective: each function most time of it's execution do nothing just waiting for network results. It would be cool if we somehow would be able to use CPU for second request while first one stuck with network stuff and don't need it.

  2. This problem can be solved (and usually solves) by running functions in different threads:

    with ThreadPoolExecutor(max_workers=2) as e:
    e.submit(request, url1)
    e.submit(request, url2)

    We would get results faster this way. While first request is stuck with network, CPU would be able to do something useful for second request in another thread.

    This is however not ideal solution: switching between threads have some cost, executing flow is more complex than in the first example.

    There should be way better.

  3. Use one function idle period to start executing another function is what asyncio in general about:

    await asyncio.gather(
    async_request(url1),
    async_request(url2),
)

    Event loop manages execution flow: when first coroutine reaches some I/O operation and CPU can be used to do job elsewhere, second coroutine starts. Later event loop returns to remain executing of first coroutine.

    We get "parallel" requests and clean understandable code. Since we have parallelization in single thread, we just don't need another.

Actually, when we use asyncio threads still can be useful. If we ready to pay for them, they can help us to cast synchronous I/O functions to asynchronous very quickly:

async def async_request(url):
    loop = asyncio.get_event_loop()
    return (await loop.run_in_executor(None, request, url))

But again, it's optional and we usually can find module to make requests (and other I/O tasks) asynchronously without threads.

I didn't face with any other tasks when threads can be useful in asynchronous programs.

Upvotes: 3

Related Questions