CODEWITHSUNDEEP
pythonasynchronousasync-awaitpython-asyncio
winwin
winwin

Reputation: 1797

How to make a function behave asynchronously?

I've recently been having a run-up with asynchronous functions in Python, and I wonder how one could make a synchronous function into an asynchronous one.

For example, there is the library for translation via google api pygoogletranslation. One could most possibly wonder, how to translate many different words asynchronously. Of course, you could place it into one request, but then google api would consider it a text and treat it accordingly, which will cause incorrect results.

How could one turn this code:

from pygoogletranslation import Translator
translator = Translator()
translations = []
words = ['partying', 'sightseeing', 'sleeping', 'catering']
for word in words:
    translations.append(translator.translate(word, src='en', dest='es'))
print(translations)

Into this:

from pygoogletranslation import Translator
import asyncio
translator = Translator()
translation_tasks = []
words = ['partying', 'sightseeing', 'sleeping', 'catering']
for word in words:
    asyncio.create_task(translator.translate(word, src='en', dest='es'))
translations = asyncio.run(
    asyncio.gather(translation_tasks, return_exceptions=True) 
)
print(translations)

Considering the function translate doesn't have a built-in async implementation?

Upvotes: 0

Views: 2293

Answers (3)

Ilango Rajagopal
Ilango Rajagopal

Reputation: 316

As mentioned in other answers, calling a blocking function is useless with ayncio. In this particular case, I suggest you use google-cloud-translate, which is the official translate library from Google.

You could have done something like this in your current library:

async def do_task(word):
    return translator.translate(word, ...)

def main():
    # Create translator
    ...
    asyncio.gather(do_task(word) for word in [])

But this will just run the task in the same way without asyncio. The real gain in asyncio is that, when is something pending or waiting, it can do something else. eg, while waiting for response from server, it can send another request.

How will Python know that some work is pending? Only when the function (coroutine here) notifies the event loop via await keyword. So you definitely need to use a library that natively supports async operations. The above mentioned google-cloud-translate is such a library. You can do:

from google.cloud import translate


async def main():
    # Async-supported google translator client
    client = translate.TranslationServiceAsyncClient()
    words = ['partying', 'sightseeing', 'sleeping', 'catering']
    results = await asyncio.gather(*[client.translate_text(parent=f"projects/{project_name}", contents=[word], source_language_code="en", target_language_code="es") for word in words])
    print(results)

asyncio.run(main())

You can see that this client actually takes list of strings as input, so you could directly pass the list of strings here. According to docs, the limit for that is 1024. So if your list is bigger, you have to use this for loop.

You might have to set up credentials etc for this client though, which is outside the scope of this question.

Upvotes: 2

user4815162342
user4815162342

Reputation: 154911

To make a function async, you need to define it with async def and change it to use other async functions for anything that might block - for example, instead of requests you'd use aiohttp, and so on. The point of the effort is that the function can then be executed by an event loop along with other such functions. Whenever an async function needs to wait for something, as signaled by the await keyword, it suspends to the event loop and gives others a chance to execute. The event loop will seamlessly coordinate concurrent execution of a possibly large number of such async functions. See e.g. this answer for more details.

If a critical blocking function that you are depending on doesn't have an async implementation, you can use run_in_executor (or, beginning with Python 3.9, asyncio.to_thread) to make it async. Note, however, that such solutions are "cheating" because they use threads under the hood, so they will not provide benefits normally associated by asyncio such as ability to scale beyond the number of threads in the thread pool, or ability to cancel execution of coroutines.

Upvotes: 1

Tim
Tim

Reputation: 3407

You will have to create an async function and then run it. Though if translate doesn't have built in async support or is blocking, using async will not make it faster. It's probably better to use multithreading/multiprocessing as suggested in the comments.

async def main():
    async def one_iteration(word):
        output.append(translator.translate(word, src='en', dest='es'))
    coros = [one_iteration(word) for word in words]
    await asyncio.gather(*coros)
asyncio.run(main())

Upvotes: 2

Related Questions