CODEWITHSUNDEEP
pythondjangosigterm
FlorianK
FlorianK

Reputation: 440

How to execute code on SIGTERM in Django request handler thread

I'm currently trying to understand the signal handling in Django when receiving a SIGTERM.

Background information

I have an application with potentially long running requests, running in a Docker container. When Docker wants to stop a container, it first sends a SIGTERM signal, waits for a while, and then sends a SIGKILL. Normally, on the SIGTERM, you stop receiving new requests, and hope that the currently running requests finish before Docker decides to send a SIGKILL. However, in my application, I want to save which requests have been tried, and find that more important than finishing the request right now. So I'd prefer for the current requests to shutdown on SIGTERM, so that I can gracefully end them (and saving their state), rather than waiting for the SIGKILL.

My attempt

My theory is that you can register a signal listener for SIGTERM, that performs a sys.exit(), so that a SystemExit exception is raised. I then want to catch that exception in my request handler, and save my state. As a first experiment I've created a mock project for the Django development server. I registered the signal in the Appconfig.ready() function:

import signal
import sys

from django.apps import AppConfig

import logging

logger = logging.getLogger(__name__)

def signal_handler(signal_num, frame):
    sys.exit()

class TesterConfig(AppConfig):
    default_auto_field = 'django.db.models.BigAutoField'
    name = 'tester'

    def ready(self):
        logger.info('starting ready')
        signal.signal(signal.SIGTERM, signal_handler)

and have created a request handler that catches Exceptions and BaseExceptions:

import logging
import sys
import time

from django.http import HttpResponse

def handler(request):
    try:
        logger.info('start')
        while True:
            time.sleep(1)
    except Exception:
        logger.info('exception')
    except BaseException:
        logger.info('baseexception')

    return HttpResponse('hallo')

But when I start the development server user python manage.py runserver and then send a kill signal using kill -n 15 <pid>, no 'baseexception' message gets logged ('start' does get logged). The full code can be foud here.

My question

My hypothesis is that the SIGTERM signal is handled in the main thread, so the sys.exit() call is handled in the main thread. So the exception is not raised in the thread running the request handler, and nothing is caught. How do I change my code to have the SystemError raised in the request handler thread? I need some information from that thread to log, so I can't just 'log' something in the signal handler directly.

Upvotes: 5

Views: 2524

Answers (1)

FlorianK
FlorianK

Reputation: 440

Ok, I did some investigation, and I found an answer to my own question. As I somewhat suspected while posing the question, it is the kind of question where you probably want a different solution to the one being asked for. However, I'll post my findings here, for if someone in the future finds the question and finds him- and/or herself in a similar situation.

There were a couple of reasons that the above did not work. The first is that I forgot to register my app in the INSTALLED_APPS, so the code in TesterConfig.ready was not actually executed.

Next, it turns out that Django also registers a handler for the SIGTERM signal, see the Django source code. So if you send a SIGTERM to the process, this is the one that gets triggered. I temporarily commented the line in my virtual environment to investigate some more, but of course that can never lead to a real solution.

The sys.exit() function indeed raises a SystemExit exception, but that is handled only in the thread itself. If you would want to communicate between threads, you'll probably want to use an Event and check it regularly in the thread that you want to execute.

If you're looking for suggestions how to do something like this when running Django through gunicorn, I found that if you use the sync worker, you can register signals in your views.py, because the requests will be handled in the main thread. In the end I ended up registering the signal here, and writing a logging line and raising an Exception in the signal handler. This is then handled by the exception handling that was already in place.

Upvotes: 5

Related Questions