CODEWITHSUNDEEP
python-asynciopython-3.9
wiseboar
wiseboar

Reputation: 335

Asyncio with memory leak (Python)

I have an issue with my (dockerized) application consuming an ever increasing amount of memory until the Linux kernel kills is. In the container's docker logs I only got an ominous Killed message without further context - only after checking the kernel logs (cat /var/log/kern.log) and docker stats did I realize what was happening - memory usage was going up by ~10MB per second.

The application is an asyncronous grpc-server with several concurrent tasks (using return ensure_future(self._continuous_function) to keep the function's task up and running, async so the server endpoints are not blocked).

Upvotes: 2

Views: 7940

Answers (2)

Greg
Greg

Reputation: 1719

You could schedule a task in following way this way you can ensure that the task cleanup after itself.

import asyncio
from typing import Coroutine

from marie.helper import get_or_reuse_loop

background_tasks = set()


def run_background_task(coroutine: Coroutine) -> asyncio.Task:
    """Schedule a task reliably to the event loop.

    This API is used when you don't want to cache the reference of `asyncio.Task`.
    For example,

    ```
    get_event_loop().create_task(coroutine(*args))
    ```

    The above code doesn't guarantee to schedule the coroutine to the event loops

    When using create_task in a  "fire and forget" way, we should keep the references
    alive for the reliable execution. This API is used to fire and forget
    asynchronous execution.

    https://docs.python.org/3/library/asyncio-task.html#creating-tasks
    """
    task = get_or_reuse_loop().create_task(coroutine)
    # Add task to the set. This creates a strong reference.
    background_tasks.add(task)

    # To prevent keeping references to finished tasks forever,
    # make each task remove its own reference from the set after
    # completion:
    task.add_done_callback(background_tasks.discard)
    return task

Upvotes: 0

wiseboar
wiseboar

Reputation: 335

I figured out that ensure_future() caused my memory leak issue - or rather the return I used with it. Apparently the return meant that a reference to the original task was being kept in memory (instead of being garbage collected), and the function only had a very short wait time associated with it (1ms) - so memory kept building up fast.

So after removing the leak my app now consumes about 60MB of very constant memory. I hope this helps somebody else out, I couldn't find a reference to this exact issue which is why I'm sharing it here.

Upvotes: 4

Related Questions