Django files upload causes memory leak

Question

I have an observation about memory leak:

• If we try to POST files to Django view(I use DRF's APIView), Django will not release memory even a response has been made. The memory will keep rising if you keep uploading files even if you do nothing in views.py. (I have tried uploading 100 files concurrently calling the same API). I know that if an uploaded file is smaller than 2.5 megabytes, Django will hold the entire contents of the upload in memory. So I am wondering this could be an issue for a possible memory leak because it will never release the memory.

PS: Even if I set FILE_UPLOAD_MAX_MEMORY_SIZE=0, which means use disk to store the files instead of using memory, I can still see a memory rise at the background.

I am using:

• Python3.6/Django2.1, Gunicorn
• DEBUG = False

Does anyone know how to solve this issue ? Thanks a lot.

Answer 1

It turns out to be the problem of Gunicorn. Once setting the --thread, it seems like Gunicorn somehow won't release memory until all threads were used(Not sure if it is the correct term).

This could be helpful: Gunicorn Workers and Threads

Answer 2

It is impossible to tell what is causing the problem but here is a list of things you can try:

• Ensure you are not using DEBUG=TRUE
• Try upgrading your Python/Django versions if possible
• What WSGI server are you using? (hopefully not the development server). Upgrade to the newest version or try a different one and compare (for example if you are using gunicorn try uwsgi or the opposite)
• Not a solution but a workouround: gunicorn has a max_requests setting and you specify the number of requests before restarting the worker processes. Most servers have such a setting. Restarting will free memory if its leaking.

If the issue persists you may need to trace the cause with a tool like tracemalloc.