CODEWITHSUNDEEP
pythonsignalsgunicornsigterm
Ailothaen
Ailothaen

Reputation: 91

How to handle correctly SIGTERM in a gunicorn app?

I have a Python Flask app:

from flask import Flask, Response
import signal

app = Flask("myapp")

def exit_gracefully(*args):
    print("Doing something here")

signal.signal(signal.SIGINT, exit_gracefully)
signal.signal(signal.SIGTERM, exit_gracefully)

@app.route("/myendpoint")
def myendpoint():
    print("Doing something here")
    return Response("Everything is fine", status=200, mimetype="application/json")

This app is ran by gunicorn, using a classic wsgi.py file and a systemd service:

[Unit]
Description=myservice

[Service]
User=root
Group=root
WorkingDirectory=/opt/myservice
ExecStart=/opt/myservice/run.sh

[Install]
WantedBy=multi-user.target

#!/bin/bash
source env/bin/activate
env/bin/gunicorn --bind unix:gunicorn.sock --workers 1 --timeout 10 wsgi:app

I need to intercept SIGTERM signals sent to the service, because I need to perform some cleanup and interruptions (especially setting threading.Event() flags)

However, the issue is that, when I type systemctl stop myservice, the service hangs for 30 seconds before shutting down (which is probably gunicorn's graceful_timeout setting), instead of shutting down immediately, as I intend to.

When looking in the source code of gunicorn, I notice that the Worker class uses a signal handler to shutdown itself gracefully:

# gunicorn/workers/base.py:173
signal.signal(signal.SIGTERM, self.handle_exit)

My guess is that, since it is possible to use only one handler for each signal, I am overriding this handler by using my own handler. But I still need both handlers, mine for my cleanup operations to be done and the default one for the gunicorn worker to stop properly.

Something maybe worth to mention is that, when I run run.sh in my shell and send SIGINT to it (by pressing Ctrl+C), my app stopped immediately, despite gunicorn also implements an handler for it.

Any ideas on how I should proceed? I thought about calling the function handle_exit in my own handler, but since my code is "gunicorn-agnostic", I have no idea about how I should do that. Should I change the design of my app?

Upvotes: 5

Views: 3197

Answers (2)

James Addison
James Addison

Reputation: 3094

My guess is that, since it is possible to use only one handler for each signal, I am overriding this handler by using my own handler. But I still need both handlers, mine for my cleanup operations to be done and the default one for the gunicorn worker to stop properly.

The OP posted the exact cause of the problem. From the Python docs:

signal.signal(signalnum, handler)

Set the handler for signal signalnum to the function handler. handler can be a callable Python object taking two arguments (see below), or one of the special values signal.SIG_IGN or signal.SIG_DFL. The previous signal handler will be returned...

As such, you would need to capture the previous handler function (gunicorn's) when calling signal.signal(...) to set your own handler in place. When you receive the signal (INT, TERM, QUIT, etc), your handler would need to call both the previous handler function (ie. gunicorn's) as well as your own signal logic.

Upvotes: 0

Ruben
Ruben

Reputation: 1

I have exactly the same problem. When receiving SIGTERM, my worker doesn't stop its execution. In my case, I'm running a thread along with the Flask app. I was receiving the signal and successfully stopping the thread, but gunicorn didn't stop. The solution was to add an exit(0) statement after the thread.join().

I assume that without an exit code, gunicorn somehow thinks that the process is still alive.

from flask import Flask, Response
import signal
import time
import threading

app = Flask("myapp")

@app.route("/myendpoint")
def myendpoint():
    print("Doing something here")
    return Response("Everything is fine", status=200, mimetype="application/json")

must_stop = False
def sleppy_thread():
    while True:
        time.sleep(0.1)
        if must_stop:
            break

def exit_gracefully(*args):
    global must_stop
    must_stop = True

signal.signal(signal.SIGINT, exit_gracefully)
signal.signal(signal.SIGTERM, exit_gracefully)

sleepy_thread = threading.Thread(target=sleppy_thread)
sleepy_thread.start()
sleepy_thread.join()
exit(0)

Upvotes: 0

Related Questions