CODEWITHSUNDEEP
gunicorn
John
John

Reputation: 4261

gunicorn: how to resolve "WORKER TIMEOUT"?

I have setup gunicorn with 3 workers, 30 worker connections and using eventlet worker class. It is set up behind Nginx. After every few requests, I see this in the logs.

[ERROR] gunicorn.error: WORKER TIMEOUT (pid:23475)
None
[INFO] gunicorn.error: Booting worker with pid: 23514

Why is this happening? How can I figure out what's going wrong?

Upvotes: 369

Views: 443486

Answers (21)

Ramon Medeiros
Ramon Medeiros

Reputation: 2616

Is this endpoint taking too much time?

Maybe you are using flask without asynchronous support, so every request will block the call? To create async support, add the gevent worker. With gevent, a new call will spawn a new thread, and you app will be able to receive more requests simultaneously.

pip install gevent
gunicon .... --worker-class gevent

Upvotes: 30

Daniel Olson
Daniel Olson

Reputation: 531

In my case I came across this issue when sending larger(10MB) files to my server. My development server(app.run()) received them no problem but gunicorn could not handle them.

for people who come to the same problem I did. My solution was to send it in chunks like this: ref / html example, separate large files ref


    def upload_to_server():
        upload_file_path = location
    
        def read_in_chunks(file_object, chunk_size=524288):
            """Lazy function (generator) to read a file piece by piece.
            Default chunk size: 1k."""
            while True:
                data = file_object.read(chunk_size)
                if not data:
                    break
                yield data
    
        with open(upload_file_path, 'rb') as f:
            for piece in read_in_chunks(f):
                r = requests.post(
                    url + '/api/set-doc/stream' + '/' + server_file_name,
                    files={name: piece},
                    headers={'key': key, 'allow_all': 'true'})

my flask server:


    @app.route('/api/set-doc/stream/<name>', methods=['GET', 'POST'])
    def api_set_file_streamed(name):
        folder = escape(name)  # secure_filename(escape(name))
        if 'key' in request.headers:
            if request.headers['key'] != key:                
                return ''
        else:
            return ''
        for fn in request.files:
            file = request.files[fn]
            if fn == '':
                print('no file name')
                flash('No selected file')
                return 'fail'
            if file and allowed_file(file.filename):
                file_dir_path = os.path.join(app.config['UPLOAD_FOLDER'], folder)
                if not os.path.exists(file_dir_path):
                    os.makedirs(file_dir_path)
                file_path = os.path.join(file_dir_path, secure_filename(file.filename)) 
                with open(file_path, 'ab') as f:
                    f.write(file.read())
                return 'sucess'
        return ''

Upvotes: 3

Cloverr
Cloverr

Reputation: 228

Apart from the gunicorn timeout settings which are already suggested, since you are using nginx in front, you can check if these 2 parameters works, proxy_connect_timeout and proxy_read_timeout which are by default 60 seconds. Can set them like this in your nginx configuration file as,

proxy_connect_timeout 120s;
proxy_read_timeout 120s;

Upvotes: 0

YazanGhafir
YazanGhafir

Reputation: 864

The easiest way that worked for me is to create a new config.py file in the same folder where your app.py exists and to put inside it the timeout and all your desired special configuration:

timeout = 999

Then just run the server while pointing to this configuration file

gunicorn -c config.py --bind 0.0.0.0:5000 wsgi:app

note that for this statement to work you need wsgi.py also in the same directory having the following

from myproject import app

if __name__ == "__main__":
    app.run()

Cheers!

Upvotes: 0

Gena Kukartsev
Gena Kukartsev

Reputation: 1705

Check that your workers are not killed by a health check. A long request may block the health check request, and the worker gets killed by your platform because the platform thinks that the worker is unresponsive.

E.g. if you have a 25-second-long request, and a liveness check is configured to hit a different endpoint in the same service every 10 seconds, time out in 1 second, and retry 3 times, this gives 10+1*3 ~ 13 seconds, and you can see that it would trigger some times but not always.

The solution, if this is your case, is to reconfigure your liveness check (or whatever health check mechanism your platform uses) so it can wait until your typical request finishes. Or allow for more threads - something that makes sure that the health check is not blocked for long enough to trigger worker kill.

You can see that adding more workers may help with (or hide) the problem.

Upvotes: 1

Drayen D&#246;rff
Drayen D&#246;rff

Reputation: 103

in case you have changed the name of the django project you should also go to

cd /etc/systemd/system/

then

sudo nano gunicorn.service

then verify that at the end of the bind line the application name has been changed to the new application name

Upvotes: -6

Chayan Bansal
Chayan Bansal

Reputation: 2075

The Microsoft Azure official documentation for running Flask Apps on Azure App Services (Linux App) states the use of timeout as 600

gunicorn --bind=0.0.0.0 --timeout 600 application:app

https://learn.microsoft.com/en-us/azure/app-service/configure-language-python#flask-app

Upvotes: 43

Susan Enneking
Susan Enneking

Reputation: 1

Frank's answer pointed me in the right direction. I have a Digital Ocean droplet accessing a managed Digital Ocean Postgresql database. All I needed to do was add my droplet to the database's "Trusted Sources".

(click on database in DO console, then click on settings. Edit Trusted Sources and select droplet name (click in editable area and it will be suggested to you)).

Upvotes: 0

frank
frank

Reputation: 186

For me, it was because I forgot to setup firewall rule on database server for my Django.

Upvotes: 0

gwik
gwik

Reputation: 739

Run Gunicorn with --log-level debug.

It should give you an app stack trace.

Upvotes: 39

Skerrepy
Skerrepy

Reputation: 392

This worked for me:

gunicorn app:app -b :8080 --timeout 120 --workers=3 --threads=3 --worker-connections=1000

If you have eventlet add:

--worker-class=eventlet

If you have gevent add:

--worker-class=gevent

Upvotes: 16

Mao
Mao

Reputation: 48

timeout is a key parameter to this problem.

however it's not suit for me.

i found there is not gunicorn timeout error when i set workers=1.

when i look though my code, i found some socket connect (socket.send & socket.recv) in server init.

socket.recv will block my code and that's why it always timeout when workers>1

hope to give some ideas to the people who have some problem with me

Upvotes: 1

Preethi Vaidyanathan
Preethi Vaidyanathan

Reputation: 1322

For me, the solution was to add --timeout 90 to my entrypoint, but it wasn't working because I had TWO entrypoints defined, one in app.yaml, and another in my Dockerfile. I deleted the unused entrypoint and added --timeout 90 in the other.

Upvotes: 0

Haider Lasne
Haider Lasne

Reputation: 21

If you are using GCP then you have to set workers per instance type.

Link to GCP best practices https://cloud.google.com/appengine/docs/standard/python3/runtime

Upvotes: 2

Artem Zaika
Artem Zaika

Reputation: 1201

I've got the same problem in Docker.

In Docker I keep trained LightGBM model + Flask serving requests. As HTTP server I used gunicorn 19.9.0. When I run my code locally on my Mac laptop everything worked just perfect, but when I ran the app in Docker my POST JSON requests were freezing for some time, then gunicorn worker had been failing with [CRITICAL] WORKER TIMEOUT exception.

I tried tons of different approaches, but the only one solved my issue was adding worker_class=gthread.

Here is my complete config: 

import multiprocessing

workers = multiprocessing.cpu_count() * 2 + 1
accesslog = "-" # STDOUT
access_log_format = '%(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(q)s" "%(D)s"'
bind = "0.0.0.0:5000"
keepalive = 120
timeout = 120
worker_class = "gthread"
threads = 3

Upvotes: 12

hashlash
hashlash

Reputation: 1015

WORKER TIMEOUT means your application cannot response to the request in a defined amount of time. You can set this using gunicorn timeout settings. Some application need more time to response than another.

Another thing that may affect this is choosing the worker type

The default synchronous workers assume that your application is resource-bound in terms of CPU and network bandwidth. Generally this means that your application shouldn’t do anything that takes an undefined amount of time. An example of something that takes an undefined amount of time is a request to the internet. At some point the external network will fail in such a way that clients will pile up on your servers. So, in this sense, any web application which makes outgoing requests to APIs will benefit from an asynchronous worker.

When I got the same problem as yours (I was trying to deploy my application using Docker Swarm), I've tried to increase the timeout and using another type of worker class. But all failed.

And then I suddenly realised I was limitting my resource too low for the service inside my compose file. This is the thing slowed down the application in my case

deploy:
  replicas: 5
  resources:
    limits:
      cpus: "0.1"
      memory: 50M
  restart_policy:
    condition: on-failure

So I suggest you to check what thing slowing down your application in the first place

Upvotes: 19

Apurv Agarwal
Apurv Agarwal

Reputation: 3188

On Google Cloud Just add --timeout 90 to entrypoint in app.yaml

entrypoint: gunicorn -b :$PORT main:app --timeout 90

Upvotes: 85

Ranc
Ranc

Reputation: 183

Could it be this? http://docs.gunicorn.org/en/latest/settings.html#timeout

Other possibilities could be your response is taking too long or is stuck waiting.

Upvotes: 17

James Lin
James Lin

Reputation: 26528

I had very similar problem, I also tried using "runserver" to see if I could find anything but all I had was a message Killed

So I thought it could be resource problem, and I went ahead to give more RAM to the instance, and it worked.

Upvotes: 7

Amit Talmor
Amit Talmor

Reputation: 8174

We had the same problem using Django+nginx+gunicorn. From Gunicorn documentation we have configured the graceful-timeout that made almost no difference.

After some testings, we found the solution, the parameter to configure is: timeout (And not graceful timeout). It works like a clock..

So, Do:

1) open the gunicorn configuration file

2) set the TIMEOUT to what ever you need - the value is in seconds

NUM_WORKERS=3
TIMEOUT=120

exec gunicorn ${DJANGO_WSGI_MODULE}:application \
--name $NAME \
--workers $NUM_WORKERS \
--timeout $TIMEOUT \
--log-level=debug \
--bind=127.0.0.1:9000 \
--pid=$PIDFILE

Upvotes: 356

Dseed
Dseed

Reputation: 172

You need to used an other worker type class an async one like gevent or tornado see this for more explanation : First explantion :

You may also want to install Eventlet or Gevent if you expect that your application code may need to pause for extended periods of time during request processing

Second one :

The default synchronous workers assume that your application is resource bound in terms of CPU and network bandwidth. Generally this means that your application shouldn’t do anything that takes an undefined amount of time. For instance, a request to the internet meets this criteria. At some point the external network will fail in such a way that clients will pile up on your servers.

Upvotes: 8

Related Questions