uwsgi working with nginx, systemd emperor not working with same configs and setup

Question

The issue here is that everything is configured, and runs without error, and yet I do not know why it is not working. As you'll see I get no app can't load error, I get no errors at all, in fact it's the most complete without errors I've had since I started. And yet, 500 response. Alot of this is here for completeness as I'll explain. If I take emperor out of the mix, it works, just fine.

I have been trying to deploy an opensource flask app ceph-dash for monitoring via uwsgi/nginx. I succeeded eventually albeit painfully. I have written a gist about my success https://gist.github.com/Lighiche/a6aec14166d62b4f8f013415a2c1f757

Howver, since there's no daemon management and the likes I thought it best to add to what I've done by using uwsgi emperor and I followed this guide to the letter https://chriswarrick.com/blog/2016/02/10/deploying-python-web-apps-with-nginx-and-uwsgi-emperor/

I am on Centos 7. I have successful run and tested the app on this implementation as you can see in the gist, I actually went out of my way to discover how to configure uwsgi under different scenarios.

However with the emperor implementation of wsgi and all it's config files. I am getting a 500 response internal server error. And I think it's to do with the hand off between nginx and emperor or emperor itself. As when I stop the uwsgi service I get a 502. So nginx at least sees the socket file.

WSGI server start log, all green

*** Starting uWSGI 2.0.13.1 (64bit) on [Thu Sep  1 11:16:24 2016] ***
compiled with version: 4.8.5 20150623 (Red Hat 4.8.5-4) on 02 August 2016 21:07:54
os: Linux-3.10.0-327.28.3.el7.x86_64 #1 SMP Thu Aug 18 19:05:49 UTC 2016
nodename: prdceph-mon00
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 2
current working directory: /etc/uwsgi.d
detected binary path: /usr/sbin/uwsgi
chdir() to /etc/nginx/sites-enabled/ceph-dash
your processes number limit is 7282
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address /var/run/uwsgi/ceph-dash.sock fd 6
Python version: 2.7.5 (default, Aug 18 2016, 15:58:25)  [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)]
Python main interpreter initialized at 0xee1030
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 363840 bytes (355 KB) for 4 cores
*** Operational MODE: preforking ***
WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0xee1030 pid: 3236 (default app)
mountpoint  already configured. skip.
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 3236)
spawned uWSGI worker 1 (pid: 3241, cores: 1)
spawned uWSGI worker 2 (pid: 3242, cores: 1)
spawned uWSGI worker 3 (pid: 3243, cores: 1)
spawned uWSGI worker 4 (pid: 3244, cores: 1)
[pid: 3243|app: 0|req: 1/1] 127.0.0.1 () {34 vars in 429 bytes} [Thu Sep  1 11:17:31 2016] GET / => generated 291 bytes in 77 msecs (HTTP/1.1 500) 2 headers in 84 bytes (1 switches on core 0)
announcing my loyalty to the Emperor...

WSGI service start log, all green

 uwsgi.service - uWSGI Emperor Service
   Loaded: loaded (/usr/lib/systemd/system/uwsgi.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2016-09-01 11:16:24 BST; 2s ago
  Process: 3231 ExecStartPre=/bin/chown uwsgi:uwsgi /run/uwsgi (code=exited, status=0/SUCCESS)
  Process: 3229 ExecStartPre=/bin/mkdir -p /run/uwsgi (code=exited, status=0/SUCCESS)
 Main PID: 3234 (uwsgi)
   Status: "The Emperor is governing 1 vassals"
   CGroup: /system.slice/uwsgi.service
           ├─3234 /usr/sbin/uwsgi --ini /etc/uwsgi.ini
           ├─3235 /usr/sbin/uwsgi --ini /etc/uwsgi.ini
           ├─3236 /usr/sbin/uwsgi --ini cephdash.ini
           ├─3241 /usr/sbin/uwsgi --ini cephdash.ini
           ├─3242 /usr/sbin/uwsgi --ini cephdash.ini
           ├─3243 /usr/sbin/uwsgi --ini cephdash.ini
           └─3244 /usr/sbin/uwsgi --ini cephdash.ini

Sep 01 11:16:24 prdceph-mon00 uwsgi[3234]: thunder lock: disabled (you can enable it with --thunder-lock)
Sep 01 11:16:24 prdceph-mon00 uwsgi[3234]: your mercy for graceful operations on workers is 60 seconds
Sep 01 11:16:24 prdceph-mon00 uwsgi[3234]: *** Operational MODE: no-workers ***
Sep 01 11:16:24 prdceph-mon00 uwsgi[3234]: spawned uWSGI master process (pid: 3234)
Sep 01 11:16:24 prdceph-mon00 uwsgi[3234]: *** Stats server enabled on /run/uwsgi/stats.sock fd: 7 ***
Sep 01 11:16:24 prdceph-mon00 uwsgi[3234]: *** starting uWSGI Emperor ***
Sep 01 11:16:24 prdceph-mon00 uwsgi[3234]: *** has_emperor mode detected (fd: 7) ***
Sep 01 11:16:24 prdceph-mon00 uwsgi[3234]: [uWSGI] getting INI configuration from cephdash.ini
Sep 01 11:16:25 prdceph-mon00 uwsgi[3234]: Thu Sep  1 11:16:25 2016 - [emperor] vassal cephdash.ini has been spawned
Sep 01 11:16:25 prdceph-mon00 uwsgi[3234]: Thu Sep  1 11:16:25 2016 - [emperor] vassal cephdash.ini is ready to accept requests

WSGI server response 500

[pid: 3243|app: 0|req: 1/1] 127.0.0.1 () {34 vars in 429 bytes} [Thu Sep  1 11:17:31 2016] GET / => generated 291 bytes in 77 msecs (HTTP/1.1 500) 2 headers in 84 bytes (1 switches on core 0)

NGINX server response 500

<ip> - - [01/Sep/2016:11:14:16 +0100] "GET / HTTP/1.1" 500 291 "http://<ip>/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36" "<my ip>, <ip>"
127.0.0.1 - - [01/Sep/2016:11:17:31 +0100] "GET / HTTP/1.1" 500 291 "-" "Python-urllib/2.7" 

cat /etc/uwsgi.ini

[uwsgi]
uid = uwsgi
gid = nginx
pidfile = /run/uwsgi/uwsgi.pid
emperor = /etc/uwsgi.d
stats = /run/uwsgi/stats.sock
#emperor-tyrant = true
cap = setgid,setuid

cat /etc/uwsgi.d/cephdash.ini (this is a symlink to the wsgi.ini for the app)

[uwsgi]
chdir           = /etc/nginx/sites-enabled/ceph-dash
wsgi-file       = /etc/nginx/sites-enabled/ceph-dash/contrib/wsgi/cephdash.wsgi
module          = ceph-dash:app
enable-threads  = true
master          = true
processes       = 4
socket          = /var/run/uwsgi/ceph-dash.sock
chmod-socket    = 777
vacuum          = true
uid             = uwsgi
gid             = nginx
daemonize       = /tmp/ceph-dash.log
plugins         = python,logfile
logger          = file:/tmp/myappuwsgi.log

cat /etc/nginx/conf.d/cephdash.conf

upstream uwsgi {
    server unix:///var/run/uwsgi/ceph-dash.sock;
}

server {
    listen 5000;
    server_name <hostname>
    charset     utf-8;

    location / {
        uwsgi_pass  uwsgi;
        include     /etc/nginx/sites-enabled/ceph-dash/contrib/nginx/uwsgi_params; 
    }
}

---

I have uninstalled the uwsgi that I installed via pip. I appear still to have a /sbin/uwsgi (linked from /usr/sbin) which looks to be the one installed for systemd and a /usr/bin/uwsgi I don't know how that happened. I changed the .service file to use /usr/bin and have the same error. However the binary from the rpm does not have the plugsin installed and the one in /usr/bin does. systemd uwsgi now starts with /usr/bin/uwsgi and as you can see below that binary loads the plugins required but I get the same internal server error, and the same messaged from systemd and startup logs no errors, no failures. Just a 500.

rpm -ql uwsgi
/etc/uwsgi.d
/etc/uwsgi.ini
/run/uwsgi
/usr/lib/systemd/system/uwsgi.service
/usr/sbin/uwsgi
/usr/share/doc/uwsgi-2.0.13.1
/usr/share/doc/uwsgi-2.0.13.1/CHANGELOG
/usr/share/doc/uwsgi-2.0.13.1/README
/usr/share/doc/uwsgi-2.0.13.1/README.Fedora
/usr/share/licenses/uwsgi-2.0.13.1
/usr/share/licenses/uwsgi-2.0.13.1/LICENSE

python --version
Python 2.7.5
uwsgi --version
2.0.13.1
which uwsgi (POINTING TO THE RIGHT PLACE but systemd is now loading /usr/bin)
/sbin/uwsgi

uwsgi --plugins-list (DOES NOT SHOW THE PLUGINS LOADED)

*** uWSGI loaded generic plugins ***
corerouter

*** uWSGI loaded request plugins ***
100: ping
101: echo
--- end of plugins list ---

*** Starting uWSGI 2.0.13.1 (64bit) on [Thu Sep  8 09:58:40 2016] ***
compiled with version: 4.8.5 20150623 (Red Hat 4.8.5-4) on 02 August 2016 21:07:54
os: Linux-3.10.0-327.28.3.el7.x86_64 #1 SMP Thu Aug 18 19:05:49 UTC 2016
nodename: prdceph-mon00
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 2
current working directory: /tmp
detected binary path: /usr/sbin/uwsgi
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
*** WARNING: you are running uWSGI without its master process manager ***
your processes number limit is 7282
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
The -s/--socket option is missing and stdin is not a socket.


/usr/bin/uwsgi --plugins-list
DOES SHOW THE PLUGINS LOADED
*** uWSGI loaded generic plugins ***
gevent
nagios
rrdtool
carbon
corerouter
fastrouter
http
ugreen
syslog
rsyslog
logsocket
router_uwsgi
router_redirect
router_basicauth
zergpool
redislog
mongodblog
router_rewrite
router_http
logfile
router_cache
rawrouter
router_static
sslrouter
cheaper_busyness
transformation_tofile
transformation_gzip
transformation_chunked
transformation_offload
router_memcached
router_redis
router_hash
router_expires
router_metrics
transformation_template
stats_pusher_socket

*** uWSGI loaded request plugins ***
0: python
17: spooler
18: symcall
100: ping
110: signal
111: cache
173: rpc
--- end of plugins list ---

*** Starting uWSGI 2.0.13.1 (64bit) on [Thu Sep  8 09:58:50 2016] ***
compiled with version: 4.8.5 20150623 (Red Hat 4.8.5-4) on 29 August 2016 09:55:26
os: Linux-3.10.0-327.28.3.el7.x86_64 #1 SMP Thu Aug 18 19:05:49 UTC 2016
nodename: prdceph-mon00
machine: x86_64
clock source: unix
detected number of CPU cores: 2
current working directory: /tmp
detected binary path: /usr/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
*** WARNING: you are running uWSGI without its master process manager ***
your processes number limit is 7282
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
The -s/--socket option is missing and 

Answer 1

For me, I tried to restart nginx but it kept giving an error. I found the problem to be an extra .conf file in sitesavailable folder.

Answer 2

uWSGI installed by yum and by pip are different. Systemd will probably use one installed by yum. When you're SSH'ing and running uWSGI by yourself, system will use one installed by pip by default.

Main difference between that uWSGI versions is: lack of python support in one installed by yum, because it tends to be modular. You can install it, probably by using yum install uwsgi-python (check your distribution repositories for specific package name). Be aware that different python plugins are needed for different python versions (2.7 will require different plugin than 3.5) and adjust package name to be installed accordingly.

After installation of python plugin check that's plugin name (visible for uWSGI) and adjust value in plugins in your .ini file. That name will probably match installed package name without uwsgi- prefix.