CODEWITHSUNDEEP
pythonmultithreadingasynchronouspython-asyncio
Asher11
Asher11

Reputation: 1325

python & db async requests (aka fire-and-forget): how to?

in the middle of a function I'd like to be able to fire off a call to the DB (burn results) but the function keeps on running so that I do not experience an I/O bottleneck. this is NOT a web application. all is offline.

snippet for explanatory purposes:

a = list(range(100))
for i in a:
    my_output = very_long_function(i)
    # I'd like to_sql to run in a "fire-and-forget fashion"
    function_of_choice_to_sql(my_output)

I was wondering whether I was better off with the threading library, asyncio or other tools. I was unsuccessful in this particular endeavour with none of them. i'll take any working solution.

any help?

p.s.: there will like be no problems with concurrency/locking and the like since in my case the time my function takes to compute is far larger than the time for the database to be written.

Upvotes: 1

Views: 451

Answers (1)

Vincent
Vincent

Reputation: 13415

You could use a ThreadPoolExecutor, it provides a simple interface to schedule callables to a pool of worker. In particular, you might be interested in the map method:

from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor(max_workers=4) as executor:
    # Lazy map (python3 only)
    outputs = map(very_long_function, range(10)) 
    # Asynchronous map
    results = executor.map(function_of_choice_to_sql, outputs)
    # Wait for the results
    print(list(results))

Upvotes: 2

Related Questions