CODEWITHSUNDEEP
nginxflaskgunicornsupervisord
BradleyMoore
BradleyMoore

Reputation: 145

nginx only passing base '/' url to gunicorn

I'm using nginx to pass requests to gunicorn for my flask app. For some reason my site example.com routes correctly but nothing else will, including example.com/index or example.com/bot/callback both of which are set up in my views.py. All routes other that example.com return a 404.

I believe it's an issue with my nginx setup because the nginx access logs register my requests but gunicorns access logs only show requests to '/'.

/etc/nginx/sites-available/example.com (based on gunicorn's example)

worker_processes 1;

user www-data www-data;
# 'user nobody nobody;' for systems with 'nobody' as a group instead
pid /tmp/nginx.pid;
access_log  /var/www/example.com/logs/nginx_access combined;
error_log   /var/www/example.com/logs/nginx_error;

events {
  worker_connections 1024; # increase if you have lots of clients
  accept_mutex off; # set to 'on' if nginx worker_processes > 1
  # 'use epoll;' to enable for Linux 2.6+
  # 'use kqueue;' to enable for FreeBSD, OSX
}

http {
  include mime.types;
  # fallback in case we can't determine a type
  default_type application/octet-stream;
  sendfile on;

  upstream app_server {
    # fail_timeout=0 means we always retry an upstream even if it failed
    # to return a good HTTP response

    # for UNIX domain socket setups
    server unix:/tmp/gunicorn.sock fail_timeout=0;

    # for a TCP configuration
    # server 192.168.0.7:8000 fail_timeout=0;
  }

  server {
    # if no Host match, close the connection to prevent host spoofing
    listen 80 default_server;
    return 444;
  }

  server {
    # use 'listen 80 deferred;' for Linux
    # use 'listen 80 accept_filter=httpready;' for FreeBSD
    listen 80;
    client_max_body_size 4G;

    # set the correct host(s) for your site
    server_name example.com www.example.com;

    keepalive_timeout 5;

    # path for static files
    root /var/www/example.com/public_html/app/static;

    location / {
      # checks for static file, if not found proxy to app
      try_files $uri @proxy_to_app;
    }

    location @proxy_to_app {
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      # enable this if and only if you use HTTPS
      # proxy_set_header X-Forwarded-Proto https;
      proxy_set_header Host $http_host;
      # we don't want nginx trying to do something clever with
      # redirects, we set the Host: header above already.
      proxy_redirect off;
      proxy_pass http://127.0.0.1:8001;
    }

    #error_page 500 502 503 504 /500.html;
    #location = /500.html {
    #  root /path/to/app/current/public;
    #}
  }
}

/var/www/example.com/gunicorn.conf (also based on gunicorn's example)

# basic setup
name = 'example.com'
user = 'www-data'
group = 'www-data'
bind = '127.0.0.1:8001'
backlog = 2048
pythonpath = '/var/www/example.com'

# workers
workers = 3
worker_class = 'sync'
worker_connections = 1000
timeout = 30
keepalive = 2
backlog = 2048

# logging
loglevel = 'info'
accesslog = '/var/www/example.com/logs/gunicorn_access'
errorlog = '/var/www/example.com/logs/gunicorn_error'
spew = 'False'

# functions
def post_fork(server, worker):
    server.log.info("Worker spawned (pid: %s)", worker.pid)

def pre_fork(server, worker):
    pass

def pre_exec(server):
    server.log.info("Forked child, re-executing.")

def when_ready(server):
    server.log.info("Server is ready. Spawning workers")

def worker_int(worker):
    worker.log.info("worker received INT or QUIT signal")

    ## get traceback info
    import threading, sys, traceback
    id2name = dict([(th.ident, th.name) for th in threading.enumerate()])
    code = []
    for threadId, stack in sys._current_frames().items():
        code.append("\n# Thread: %s(%d)" % (id2name.get(threadId,""),
            threadId))
        for filename, lineno, name, line in traceback.extract_stack(stack):
            code.append('File: "%s", line %d, in %s' % (filename,
                lineno, name))
            if line:
                code.append("  %s" % (line.strip()))
    worker.log.debug("\n".join(code))

def worker_abort(worker):
    worker.log.info("worker received SIGABRT signal")

/var/www/example.com/public_html/app/__init__.py

from flask import Flask
import logging
from werkzeug.contrib.fixers import ProxyFix


app = Flask(__name__)
app.config.from_pyfile('config.py')
app.wsgi_app = ProxyFix(app.wsgi_app)

logging.basicConfig(filename='../../logs/flask', level=logging.DEBUG)

import views


if __name__ == '__main__':
    app.run()

/var/www/example.com/public_html/app/views.py

from flask import abort, flash, jsonify, redirect, render_template, request, url_for

from __init__ import app
from automate import activate



@app.route('/')
@app.route('/index')
def index():
    return 'Example Index'


@app.route('/bot/callback', methods=['GET', 'POST'])
def bot_callback():
    #callback = get_callback(request.json)
    activate(request.json)
    return 'SuccessBot'

Not sure if it matters but I have supervisor starting gunicorn.

/etc/supervisor/conf.d/example.com.conf

[program:example.com]
command=/home/user/.virtualenvs/example.com/bin/gunicorn -c /var/www/example.com/gunicorn.conf __init__:app
directory=/var/www/example.com/public_html/app
user=www-data
group=www-data
stdout_logfile=/var/www/example.com/logs/supervisor
autostart=true
autorestart=true
redirect_stderr=true
stopsignal=INT

What am I missing here? Thanks for your help.

Update 1:

'/var/www/example.com/logs/nginx_access'

216.248.228.41 - - [17/Dec/2015:11:43:06 -0700] "GET / HTTP/1.1" 200 51 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"
216.248.228.41 - - [17/Dec/2015:11:43:11 -0700] "GET /index HTTP/1.1" 404 208 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"
216.248.228.41 - - [17/Dec/2015:11:43:16 -0700] "GET /bot/callback HTTP/1.1" 404 208 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"
216.248.228.41 - - [17/Dec/2015:11:43:19 -0700] "GET / HTTP/1.1" 200 51 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"
216.248.228.41 - - [17/Dec/2015:12:04:12 -0700] "GET / HTTP/1.1" 200 51 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"
216.248.228.41 - - [17/Dec/2015:12:04:18 -0700] "GET /bot/callback HTTP/1.1" 404 208 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"
216.248.228.41 - - [17/Dec/2015:12:04:22 -0700] "GET / HTTP/1.1" 200 51 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"
216.248.228.41 - - [17/Dec/2015:12:14:45 -0700] "GET /index HTTP/1.1" 404 208 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"
216.248.228.41 - - [17/Dec/2015:12:14:58 -0700] "GET / HTTP/1.1" 200 51 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"
216.248.228.41 - - [17/Dec/2015:12:57:21 -0700] "GET / HTTP/1.1" 200 51 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"
216.248.228.41 - - [17/Dec/2015:12:57:21 -0700] "GET /favicon.ico HTTP/1.1" 404 208 "http://www.example.com/" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"
216.248.228.41 - - [17/Dec/2015:12:57:26 -0700] "GET /index HTTP/1.1" 404 208 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"
216.248.228.41 - - [17/Dec/2015:12:57:30 -0700] "GET /bot/callback HTTP/1.1" 404 208 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"

'/var/www/example.com/logs/gunicorn_access'

216.248.228.41 - - [17/Dec/2015:11:43:06 -0700] "GET / HTTP/1.0" 200 20 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"
216.248.228.41 - - [17/Dec/2015:11:43:19 -0700] "GET / HTTP/1.0" 200 20 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"
216.248.228.41 - - [17/Dec/2015:12:04:12 -0700] "GET / HTTP/1.0" 200 20 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"
216.248.228.41 - - [17/Dec/2015:12:04:22 -0700] "GET / HTTP/1.0" 200 20 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"
216.248.228.41 - - [17/Dec/2015:12:14:58 -0700] "GET / HTTP/1.0" 200 20 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"
216.248.228.41 - - [17/Dec/2015:12:57:21 -0700] "GET / HTTP/1.0" 200 20 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36"

'/var/www/example.com/logs/gunicorn_error'

[2015-12-17 11:41:42 -0700] [13894] [INFO] Starting gunicorn 19.4.1
[2015-12-17 11:41:42 -0700] [13894] [INFO] Listening at: http://127.0.0.1:8001 (13894)
[2015-12-17 11:41:42 -0700] [13894] [INFO] Using worker: sync
[2015-12-17 11:41:42 -0700] [13894] [INFO] Server is ready. Spawning workers
[2015-12-17 11:41:42 -0700] [13906] [INFO] Booting worker with pid: 13906
[2015-12-17 11:41:42 -0700] [13906] [INFO] Worker spawned (pid: 13906)
[2015-12-17 11:41:42 -0700] [13908] [INFO] Booting worker with pid: 13908
[2015-12-17 11:41:42 -0700] [13908] [INFO] Worker spawned (pid: 13908)
[2015-12-17 11:41:42 -0700] [13910] [INFO] Booting worker with pid: 13910
[2015-12-17 11:41:42 -0700] [13910] [INFO] Worker spawned (pid: 13910)
[2015-12-17 12:03:04 -0700] [13894] [INFO] Handling signal: int
[2015-12-17 12:03:04 -0700] [13906] [INFO] worker received INT or QUIT signal
[2015-12-17 12:03:04 -0700] [13908] [INFO] worker received INT or QUIT signal
[2015-12-17 12:03:04 -0700] [13910] [INFO] worker received INT or QUIT signal
[2015-12-17 12:03:04 -0700] [13908] [INFO] Worker exiting (pid: 13908)
[2015-12-17 12:03:04 -0700] [13906] [INFO] Worker exiting (pid: 13906)
[2015-12-17 12:03:04 -0700] [13910] [INFO] Worker exiting (pid: 13910)
[2015-12-17 12:03:04 -0700] [13894] [INFO] Shutting down: Master
[2015-12-17 12:03:10 -0700] [14034] [INFO] Starting gunicorn 19.4.1
[2015-12-17 12:03:10 -0700] [14034] [INFO] Listening at: http://127.0.0.1:8001 (14034)
[2015-12-17 12:03:10 -0700] [14034] [INFO] Using worker: sync
[2015-12-17 12:03:10 -0700] [14034] [INFO] Server is ready. Spawning workers
[2015-12-17 12:03:10 -0700] [14040] [INFO] Booting worker with pid: 14040
[2015-12-17 12:03:10 -0700] [14040] [INFO] Worker spawned (pid: 14040)
[2015-12-17 12:03:10 -0700] [14043] [INFO] Booting worker with pid: 14043
[2015-12-17 12:03:10 -0700] [14043] [INFO] Worker spawned (pid: 14043)
[2015-12-17 12:03:10 -0700] [14044] [INFO] Booting worker with pid: 14044
[2015-12-17 12:03:10 -0700] [14044] [INFO] Worker spawned (pid: 14044)
[2015-12-17 12:57:03 -0700] [14034] [INFO] Handling signal: int
[2015-12-17 12:57:03 -0700] [14040] [INFO] worker received INT or QUIT signal
[2015-12-17 12:57:03 -0700] [14043] [INFO] worker received INT or QUIT signal
[2015-12-17 12:57:03 -0700] [14044] [INFO] worker received INT or QUIT signal
[2015-12-17 12:57:03 -0700] [14040] [INFO] Worker exiting (pid: 14040)
[2015-12-17 12:57:03 -0700] [14044] [INFO] Worker exiting (pid: 14044)
[2015-12-17 12:57:03 -0700] [14043] [INFO] Worker exiting (pid: 14043)
[2015-12-17 12:57:03 -0700] [14034] [INFO] Shutting down: Master
[2015-12-17 12:57:09 -0700] [14420] [INFO] Starting gunicorn 19.4.1
[2015-12-17 12:57:09 -0700] [14420] [INFO] Listening at: http://127.0.0.1:8001 (14420)
[2015-12-17 12:57:09 -0700] [14420] [INFO] Using worker: sync
[2015-12-17 12:57:09 -0700] [14420] [INFO] Server is ready. Spawning workers
[2015-12-17 12:57:09 -0700] [14426] [INFO] Booting worker with pid: 14426
[2015-12-17 12:57:09 -0700] [14426] [INFO] Worker spawned (pid: 14426)
[2015-12-17 12:57:09 -0700] [14429] [INFO] Booting worker with pid: 14429
[2015-12-17 12:57:09 -0700] [14429] [INFO] Worker spawned (pid: 14429)
[2015-12-17 12:57:09 -0700] [14431] [INFO] Booting worker with pid: 14431
[2015-12-17 12:57:09 -0700] [14431] [INFO] Worker spawned (pid: 14431)

All other logs are empty, including nginx_error. All the workers exits are because I either restarted or stopped and started the services.

Update 2

Apparently my site is somehow still sending nginx errors to /var/log/nginx/error.log

/var/log/nginx/error.log

2015/12/17 12:56:57 [emerg] 14404#0: "worker_processes" directive is not allowed here in /etc/nginx/sites-enabled/example.com:1

Upvotes: 2

Views: 2095

Answers (2)

BradleyMoore
BradleyMoore

Reputation: 145

All the junk at the top of my nginx config

worker_processes 1;

user www-data www-data;
# 'user nobody nobody;' for systems with 'nobody' as a group instead
pid /tmp/nginx.pid;
access_log  /var/www/example.com/logs/nginx_access combined;
error_log   /var/www/example.com/logs/nginx_error;

events {
  worker_connections 1024; # increase if you have lots of clients
  accept_mutex off; # set to 'on' if nginx worker_processes > 1
  # 'use epoll;' to enable for Linux 2.6+
  # 'use kqueue;' to enable for FreeBSD, OSX
}

http {
  include mime.types;
  # fallback in case we can't determine a type
  default_type application/octet-stream;
  sendfile on;

  upstream app_server {
    # fail_timeout=0 means we always retry an upstream even if it failed
    # to return a good HTTP response

    # for UNIX domain socket setups
    server unix:/tmp/gunicorn.sock fail_timeout=0;

    # for a TCP configuration
    # server 192.168.0.7:8000 fail_timeout=0;
  }

should have been in my nginx.conf not my specific site config.

this next part needed deletion

server {
  # if no Host match, close the connection to prevent host spoofing
  listen 80 default_server;
  return 444;
}

now i'm just getting regular old flask errors! lol

Upvotes: 1

YPCrumble
YPCrumble

Reputation: 28712

These questions are always difficult to figure out unless you provide any log messages in your nginx or gunicorn log. You may quickly find the issue there.

It sounds like nginx is working, but isn't proxying requests properly to your gunicorn server. I imagine the issue is in the upstream_app_server nginx directive...it doesn't look like it's properly connected to gunicorn. Try changing that to this:

upstream app_server {
    server 127.0.0.1:8001 fail_timeout=0;
}

If that's not the case, please provide contents of these files, as per the comments to your app:

/var/www/example.com/logs/nginx_error
/var/www/example.com/logs/gunicorn_error

It's quite possible you have more than one configuration error. The process is check the error logs, fix the bug, then try the app, then if the app isn't working check the error logs for the next bug :).

Upvotes: 1

Related Questions