CODEWITHSUNDEEP
pythonexceptionasync-awaittaskpython-asyncio
ahmetkca
ahmetkca

Reputation: 627

Asyncio wait handling Exception sometimes fails?

I have setup a async loop that creates list of tasks with limit.

        while current_index < len(numbers):
            if len(tasks) >= NO_CONCURRENT:
                _done, tasks = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
                task: Task = next(iter(_done))
                
                try:
                    task.result()
                except Exception as e:
                    logging.error(f"Error: occured {e}")
                    results.append(task.exception())
                else:
                    results.append(task.result())

it sometimes handles the exception successfully (logs error too) but sometimes doesn't even handle it at just prints out the exception (not logs it)

Example

2021-07-21,03:53:55.082 ERROR {base_events} [default_exception_handler] Task exception was never retrieved
future: <Task finished name='Task-16' coro=<AutoPart.get_all_info() done, defined at /mnt/d/PythonProjects/btyu-part-num-search/copy_main.py:258> exception=PartNumberNotFound('Part number 14111047 is NOT Found.')>
Traceback (most recent call last):
  File "/mnt/d/PythonProjects/btyu-part-num-search/copy_main.py", line 260, in get_all_info
    listing_container_id_num = await self.get_part_number(session)
  File "/mnt/d/PythonProjects/btyu-part-num-search/copy_main.py", line 141, in get_part_number
    raise PartNumberNotFound(self.part_number)
copy_main.PartNumberNotFound: Part number 14112106479456 is NOT Found.

Upvotes: 0

Views: 594

Answers (1)

user4815162342
user4815162342

Reputation: 154916

Your code doesn't handle the case when two tasks happen to complete in the same event loop iteration. In that case the done set returned by asyncio will have more than one element, despite "first" completed being requested. In that case you will access only the result of the first task, and the second one will be destroyed without anyone having accessed its exception, as the warning correctly remarks.

You should handle it with a loop:

while current_index < len(numbers):
    if len(tasks) >= NO_CONCURRENT:
        done, _ = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
        for task in done:
            try:
                task.result()
            except Exception as e:
                logging.error(f"Error: occured {e}")
                results.append(task.exception())
            else:
                results.append(task.result())

Also, note that the initial _ in front of a variable name is the convention to use for values you don't use, not the ones you do use.

Upvotes: 1

Related Questions