Bad Gateway 502 - Deploy Flask (python 3.5.2) using gunicorn nginx on centos 7

Question

As the title says, Im deploying an flask api on my server running centos 7. I tried following this guide on DigitalOcean with some minor differences. One is that i use conda instead of pip as my package manager. When I had setuped everything, as described bellow, and try to connect from another machine I get 502 Bad Gateway (error logs are provided at the end of this post). This is my first gunicorn/nginx deployment so any help or sugestions are welcome. I have tried to look at similar posts but with little progress.

This is what I have done so far

I have placed my project at /home/apps/myproject... and I conda is also installed at /home/apps/anaconda3....

The system service unit file I have looks like this

/etc/systemd/system/myproject.service

[Unit]
Description=Gunicorn instance to serve myproject
After=network.target

[Service]
User=apps
Group=nginx
WorkingDirectory=/home/apps/myproject
Environment="PATH=/home/apps/anaconda3/envs/myproject/bin"
ExecStart=/home/apps/anaconda3/envs/myproject/bin/gunicorn --workers 3 --bind unix:myproject.sock -m 007 wsgi

[Install]
WantedBy=multi-user.target

When I this was created I started the service

sudo systemctl start myproject
sudo systemctl enable myproject

After I modified my nginx config file to looks like this

/etc/nginx/nginx.conf

# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    server {
        listen 80;
        server_name xx.xxx.xxx.xx;

        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_pass http://unix:/home/apps/myproject/myproject.sock;
        }
    }

    server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  _;
        root         /usr/share/nginx/html;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
        }

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }

# Settings for a TLS enabled server.
#
#    server {
#        listen       443 ssl http2 default_server;
#        listen       [::]:443 ssl http2 default_server;
#        server_name  _;
#        root         /usr/share/nginx/html;
#
#        ssl_certificate "/etc/pki/nginx/server.crt";
#        ssl_certificate_key "/etc/pki/nginx/private/server.key";
#        ssl_session_cache shared:SSL:1m;
#        ssl_session_timeout  10m;
#        ssl_ciphers HIGH:!aNULL:!MD5;
#        ssl_prefer_server_ciphers on;
#
#        # Load configuration files for the default server block.
#        include /etc/nginx/default.d/*.conf;
#
#        location / {
#        }
#
#        error_page 404 /404.html;
#            location = /40x.html {
#        }
#
#        error_page 500 502 503 504 /50x.html;
#            location = /50x.html {
#        }
#    }

}

I changes the permissions according to the guid

sudo usermod -a -G apps nginx
chmod 710 /home/apps

Started the nginx service

sudo systemctl start nginx
sudo systemctl enable nginx

and also ran these command to allow trafic on http and https

sudo firewall-cmd --permanent --zone=public --add-service=http 
sudo firewall-cmd --permanent --zone=public --add-service=https
sudo firewall-cmd --reload

When I now try to access the server publicly on another machine i get 502 Bad Gateway. I tried

sudo cat /var/log/audit/audit.log | grep nginx | grep denied | audit2allow -M mynginx
sudo semodule -i mynginx.pp

as suggested on multiple posts, e.g. here, but these changes did not help. I get the following:

/var/log/nginx/error.log

2016/11/29 08:49:25 [crit] 55119#0: *8 connect() to unix:/home/apps/myproject/myproject.sock failed (13: Permission denied) while connecting to upstream, client: yyy.yy.yyy.yyy, server: xx.xx.xxx.xx, request: "GET / HTTP/1.1", upstream: "http://unix:/home/apps/myproject/myproject.sock:/", host: "xx.xx.xxx.xx"
2016/11/29 08:49:25 [crit] 55119#0: *8 connect() to unix:/home/apps/myproject/myproject.sock failed (13: Permission denied) while connecting to upstream, client: yyy.yy.yyy.yyy, server: xx.xx.xxx.xx, request: "GET /favicon.ico HTTP/1.1", upstream: "http://unix:/home/apps/myproject/myproject.sock:/favicon.ico", host: "xx.xx.xxx.xx", referrer: "http://xx.xx.xxx.xx/"

cat /var/log/audit/audit.log | grep nginx | grep denied

type=AVC msg=audit(1480345112.140:161350): avc:  denied  { search } for  pid=40475 comm="nginx" name="apps" dev="dm-2" ino=1207960576 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_dir_t:s0 tclass=dir
type=AVC msg=audit(1480345125.819:161351): avc:  denied  { search } for  pid=40475 comm="nginx" name="apps" dev="dm-2" ino=1207960576 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_dir_t:s0 tclass=dir
type=AVC msg=audit(1480345399.544:161714): avc:  denied  { search } for  pid=40475 comm="nginx" name="apps" dev="dm-2" ino=1207960576 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_dir_t:s0 tclass=dir
type=AVC msg=audit(1480345399.691:161715): avc:  denied  { search } for  pid=40475 comm="nginx" name="apps" dev="dm-2" ino=1207960576 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_dir_t:s0 tclass=dir
type=AVC msg=audit(1480345405.091:161716): avc:  denied  { search } for  pid=40475 comm="nginx" name="apps" dev="dm-2" ino=1207960576 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_dir_t:s0 tclass=dir
type=AVC msg=audit(1480345405.183:161717): avc:  denied  { search } for  pid=40475 comm="nginx" name="apps" dev="dm-2" ino=1207960576 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_dir_t:s0 tclass=dir
type=AVC msg=audit(1480345441.732:161721): avc:  denied  { search } for  pid=40475 comm="nginx" name="apps" dev="dm-2" ino=1207960576 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_dir_t:s0 tclass=dir
type=AVC msg=audit(1480345441.842:161722): avc:  denied  { search } for  pid=40475 comm="nginx" name="apps" dev="dm-2" ino=1207960576 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_dir_t:s0 tclass=dir
type=AVC msg=audit(1480345576.952:161729): avc:  denied  { search } for  pid=40475 comm="nginx" name="apps" dev="dm-2" ino=1207960576 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_dir_t:s0 tclass=dir
type=AVC msg=audit(1480345577.064:161730): avc:  denied  { search } for  pid=40475 comm="nginx" name="apps" dev="dm-2" ino=1207960576 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_dir_t:s0 tclass=dir
type=AVC msg=audit(1480346023.191:161731): avc:  denied  { search } for  pid=40475 comm="nginx" name="apps" dev="dm-2" ino=1207960576 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_dir_t:s0 tclass=dir
type=AVC msg=audit(1480346023.301:161732): avc:  denied  { search } for  pid=40475 comm="nginx" name="apps" dev="dm-2" ino=1207960576 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_dir_t:s0 tclass=dir
type=AVC msg=audit(1480346190.944:161733): avc:  denied  { search } for  pid=40475 comm="nginx" name="apps" dev="dm-2" ino=1207960576 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_dir_t:s0 tclass=dir
type=AVC msg=audit(1480346191.056:161734): avc:  denied  { search } for  pid=40475 comm="nginx" name="apps" dev="dm-2" ino=1207960576 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_dir_t:s0 tclass=dir
type=AVC msg=audit(1480347303.258:161771): avc:  denied  { search } for  pid=40475 comm="nginx" name="myproject" dev="dm-2" ino=805842944 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir
type=AVC msg=audit(1480347303.431:161772): avc:  denied  { search } for  pid=40475 comm="nginx" name="myproject" dev="dm-2" ino=805842944 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir
type=AVC msg=audit(1480347325.749:161776): avc:  denied  { search } for  pid=41569 comm="nginx" name="myproject" dev="dm-2" ino=805842944 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir
type=AVC msg=audit(1480347325.843:161777): avc:  denied  { search } for  pid=41569 comm="nginx" name="myproject" dev="dm-2" ino=805842944 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir
type=AVC msg=audit(1480347355.366:161783): avc:  denied  { search } for  pid=41605 comm="nginx" name="myproject" dev="dm-2" ino=805842944 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir
type=AVC msg=audit(1480347355.518:161784): avc:  denied  { search } for  pid=41605 comm="nginx" name="myproject" dev="dm-2" ino=805842944 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir
type=AVC msg=audit(1480347357.792:161785): avc:  denied  { search } for  pid=41605 comm="nginx" name="myproject" dev="dm-2" ino=805842944 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir
type=AVC msg=audit(1480347357.890:161786): avc:  denied  { search } for  pid=41605 comm="nginx" name="myproject" dev="dm-2" ino=805842944 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir
type=AVC msg=audit(1480347892.301:161796): avc:  denied  { search } for  pid=41764 comm="nginx" name="myproject" dev="dm-2" ino=805842944 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir
type=AVC msg=audit(1480347892.417:161797): avc:  denied  { search } for  pid=41764 comm="nginx" name="myproject" dev="dm-2" ino=805842944 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir
type=AVC msg=audit(1480403397.905:166856): avc:  denied  { search } for  pid=41764 comm="nginx" name="myproject" dev="dm-2" ino=805842944 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir
type=AVC msg=audit(1480403398.043:166857): avc:  denied  { search } for  pid=41764 comm="nginx" name="myproject" dev="dm-2" ino=805842944 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir
type=AVC msg=audit(1480404482.896:166913): avc:  denied  { search } for  pid=41764 comm="nginx" name="myproject" dev="dm-2" ino=805842944 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir
type=AVC msg=audit(1480404482.988:166914): avc:  denied  { search } for  pid=41764 comm="nginx" name="myproject" dev="dm-2" ino=805842944 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir
type=AVC msg=audit(1480404733.790:166933): avc:  denied  { write } for  pid=55119 comm="nginx" name="myproject.sock" dev="dm-2" ino=805843006 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=sock_file
type=AVC msg=audit(1480404733.898:166934): avc:  denied  { write } for  pid=55119 comm="nginx" name="myproject.sock" dev="dm-2" ino=805843006 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=sock_file
type=AVC msg=audit(1480404806.283:166935): avc:  denied  { write } for  pid=55119 comm="nginx" name="myproject.sock" dev="dm-2" ino=805843006 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=sock_file
type=AVC msg=audit(1480404806.376:166936): avc:  denied  { write } for  pid=55119 comm="nginx" name="myproject.sock" dev="dm-2" ino=805843006 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=sock_file
type=AVC msg=audit(1480405765.395:167002): avc:  denied  { write } for  pid=55119 comm="nginx" name="myproject.sock" dev="dm-2" ino=805843006 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=sock_file
type=AVC msg=audit(1480405765.507:167003): avc:  denied  { write } for  pid=55119 comm="nginx" name="myproject.sock" dev="dm-2" ino=805843006 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:user_home_t:s0 tclass=sock_file

Do any of you have an idea of what this could mean of preferrably, how to solve it.

Answer 1

After some fiddeling i grew tired of the permission error.

What I finally decided to do was to bind gunicorn to http://127.0.0.1:8000 like:

[Unit]
Description=Gunicorn instance to serve myproject
After=network.target

[Service]
User=apps
Group=nginx
WorkingDirectory=/home/apps/myproject
Environment="PATH=/home/apps/anaconda3/envs/myproject/bin"

# Old
ExecStart=/home/apps/anaconda3/envs/myproject/bin/gunicorn --workers 3 --bind unix:myproject.sock -m 007 wsgi

# New
ExecStart=/home/apps/anaconda3/envs/myproject/bin/gunicorn --workers 3 --bind 127.0.0.1 -m 007 wsgi

[Install]
WantedBy=multi-user.target

And change

proxy_pass http://unix:/home/apps/myproject/myproject.sock;

to

proxy_pass http://127.0.0.1:8000;

in my nginx config