CODEWITHSUNDEEP
pythongunicorn
sea-rob
sea-rob

Reputation: 2325

Can I make a gunicorn worker stop (and restart) AFTER responding to a request?

I have a worker process that loads a special data structure at startup, and then periodically afterwards.

However, the data structure -- a third party C++ module -- has a memory leak.

I tried using the gunicorn "max_requests" setting to have the workers expire after so many requests, which clears out their resources and reloads the data structure. But there were some nit-picky problems with that that I won't go into.

I tried adding an os._exit(0), forcing the worker to halt (and reload), but it meant that request got an error response.

What I'd like to do is signal gunicorn to send the response and THEN kill the worker, just as if the "max_requests" flag had been triggered.

Is there a mechanism to do that?

Upvotes: 0

Views: 2180

Answers (1)

espensb
espensb

Reputation: 29

The reload built-in function is useful for re-initializing modules, but in the documentation for that function (http://docs.python.org/2/library/functions.html#reload), you'll find this: "It is legal though generally not very useful to reload built-in or dynamically loaded modules, except for sys, main and builtin. In many cases, however, extension modules are not designed to be initialized more than once, and may fail in arbitrary ways when reloaded." In other words, it might not solve the problem in your case.

However, if you are able to move the calls to your third party module into a subprocess it is easy to ensure that it is reloaded every time, as you are now using the module by executing a script in a separate process. Just make a separate py file, e.g. do_something.py, where you do all your stuff you need to do with the subprocess. Then you can run this file using:

p = subprocess.Popen(
        [sys.executable, 'do_something.py', arg1, arg2 ...], 
        stdout=subprocess.PIPE
)

where 'arg1, arg2, ...' represents arguments that you may need to pass to your subprocess. If you need to read the output from your C++ module you can print it in do_something.py and read it in your application using:

p.stdout.read()

I admit that this is not a particularly smooth solution, but I think it should work.

Upvotes: 1

Related Questions