CODEWITHSUNDEEP
djangoubuntugunicorn
Hassan Baig
Hassan Baig

Reputation: 15804

Thousands of extraneous gunicorn workers

I'm using gunicorn 19.7.1 appserver with nginx reverse proxy for a Django project (Ubuntu 14.04 machine).

ps aux | grep gunicorn | grep -v grep | wc -l yields 3043 at the moment.

Whereas in /etc/init/gunicorn.conf, I've always had -w 33. Yet these extra workers persist even if I do sudo service gunicorn stop and sudo service gunicorn start.

How do I kill the extraneous workers?

How did this happen?

The worker count of 33 has always been properly configured on my busy production system.

However a few hours ago, I was trying python's multiprocessing on the server and things went south. Gunicorn workers ate up all the memory and took out the resident redis instances as well.

enter image description here

I reverted the change and have managed to get everything back online, except the memory hasn't been released and I've had to cope with these legacy gunicorn workers. What's going on?

Upvotes: 0

Views: 1458

Answers (1)

Jacob Budin
Jacob Budin

Reputation: 10003

Yet these extra workers persist even if I do sudo service gunicorn stop and sudo service gunicorn start.

service only manages service-initiated processes, so if you started Gunicorn workers outside of the service framework, these workers will continue to live even if you stop.

How do I kill the extraneous workers?

The fast way:

Run this command to list all gunicorn process IDs and terminate them, and then restart Gunicorn:

$ pkill gunicorn
$ sudo service gunicorn start

The better way:

  1. Identify your "desired" Gunicorn workers by finding the parent:

    $ sudo service gunicorn status

    Note the parent process ID. Let's say it's 123.

  2. Save a list of all the "desired" workers' PIDs:

     $ echo 123 > desired_workers
 $ pgrep -P 123 >> desired_workers

  3. Save a list of all workers' PIDs:

     $ pgrep gunicorn > all_workers

  4. Terminate the "undesired" workers:

     $ cat desired_workers all_workers | sort | uniq -u | xargs kill

Upvotes: 1

Related Questions