CODEWITHSUNDEEP
pythonpython-3.xpandaspython-asyncio
SAK
SAK

Reputation: 136

Making a pandas function asynchronous

Consider the following snippet.

import asyncio

import pandas as pd


# @asyncio.coroutine
async def read(some_id):
    '''
    '''
    # await ?, yield ?, yield from ?
    return pd.read_sql_query(f'''SELECT * FROM some_table WHERE id={some_id}''',
                                con) # how do I make this line awaitable.

I want to be able to make that function awaitable so that I can pass ids in an array and start queries simultaneously because it takes lots of time to read the table, something like.

async def main():
    '''
    '''
    alltasks = []
    for some_id in [3, 2, 5]:
        task = asyncio.create_task(read(some_id))
        alltasks.append(task)
    res = await asyncio.gather(*alltasks)

Right now this executes the queries synchronously because my function is not truly asynchronous, so how do I make it truly asynchronous. I know I can use asycio.sleep(0) but that will not execute the query, how can I achieve this without use of concurrent.future.ThreadPoolExecutor like in other answers.

Appreciate your help, thank you.

Upvotes: 1

Views: 2147

Answers (1)

Jan Wilamowski
Jan Wilamowski

Reputation: 3599

You can turn any regular function into an awaitable coroutine via asyncio.to_thread():

def read(some_id): # note: no await!
    return pd.read_sql_query(...)

async def main():
    tasks = [asyncio.to_thread(read, some_id) for some_id in [3, 2, 5]]
    res = await asyncio.gather(*tasks)

asyncio.run(main())

Upvotes: 2

Related Questions