Small service for activating nginx configurations / websites

Question

I have a backup server with cold-deployed (not running) websites.

The websites are also on multiple production servers and sometimes a number of them go down. I configured a DNS Fallback for my production servers, so if something goes wrong - requests are delivered to my backup server where they are handled with nginx + uwsgi.

When a request is delivered to the backup server nginx uses the default nginx configuration (/etc/nginx/conf.d/default_site.conf)

(because no website is running that matches the URL and because there is no symbolic link in /etc/nginx/sites-enabled

Note: I have nginx and uwsgi configurations for all websites in /etc/nginx/sites-available and /etc/uwsgi/apps-available)

I want to write a Python(or Perl or bash script) that will:

Whenever a request is made to the default nginx configuration:

1. Check if there is a matching nginx configuration in sites-available as well as a matching uwsgi process in apps-available. If so:
2. Make a link to the uwsgi record in apps-enabled and start the service
3. Make a link to the nginx record in sites-enabled and restart nginx

This is what I have so far:

My /etc/nginx/conf.d/default_site.conf:

server {
    listen 80 default_server;
    listen 443 ssl default_server;
    server_name backup.server.com;

    location / {
       include uwsgi_params;
       uwsgi_modifier1 5;

        #SOCKET for the app
        uwsgi_pass unix:///var/run/uwsgi/app/backupprocessor/socket;

        add_header Vary Accept-Encoding;
        add_header Cache-Control private;
    }

}

I have a bash script that enables a website (point 2 + 3 from my question)

#!/bin/sh

if [ $# -ne 1 ]; then
    echo "Wrong ammount of arguments supplied"
    echo "Usage: ./enable_website.sh website_name"
    exit 33
fi

WEBSITE_NAME="$1"
echo "Linking apps-available/${WEBSITE_NAME}.ini to apps-enabled/${WEBSITE_NAME}.ini"
sudo ln -s /etc/uwsgi/apps-available/${WEBSITE_NAME}.ini /etc/uwsgi/apps-enabled/${WEBSITE_NAME}.ini

echo "Linking sites-available/${WEBSITE_NAME} to sites-enabled/${WEBSITE_NAME}"
sudo ln -s /etc/nginx/sites-available/${WEBSITE_NAME} /etc/nginx/sites-enabled/${WEBSITE_NAME}


echo "Starting website ..."
sudo service uwsgi restart ${WEBSITE_NAME}
echo "Started !"

And I have also configured a uWSGI with Python Hello Word app which I wanted to use for point 1 but my approach might be wrong. (nginx even can't even start it on port 80 ...and that's where request will be coming)

Tested with:

uwsgi --http-socket 127.0.0.1:80 --wsgi-file app.py --master --processes 4 --threads 2 --stats 127.0.0.1:9191

Got:

probably another instance of uWSGI is running on the same address (127.0.0.1:80).
bind(): Address already in use [core/socket.c line 769]

Any help / advice about what are my options will be greatly appreciated! Merry Christmas !!

Answer 1

I think that you're taking wrong steps to achieve your goal. Maybe solution with activating proper uWSGI app and nginx site when request for particular site incomes, creating symling in *-enabled dir is possible, but there are better solutions.

I'm proposing solution with:

1. "one to rule them all" nginx config, that will simply pass your request to proper uWSGI app.
2. uWSGI emperor / vassals system with on-demand vassal spawning.

In that configuration, all of your projects must have static and media files treated same way (or simmilar enough, so we can use same nginx config for all of them). Or you can configure uWSGI to deal with them.

1. nginx configuration:

server {
    server_name   ~^(www\.)?(?<domain>.+)$;
    # server_name can contain any regular expression. Just remember that it should start with `~` and contain `^` and `$`
    # as you can see, we can use named capture group as an variable later
    # you can add any named capture group for later use

    root   /sites/$domain/public;

    location @default {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        include /etc/nginx/uwsgi_params;

        uwsgi_pass unix:/run/uwsgi/${domain}.sock;
        break;
    }

    location /static/ {
        try_files $uri @default;
    }

    location /media/ {
        try_files $uri @default;
    }

    location ~* ^/(android-(?:chrome|icon)[-0-9x]*\.png|ms(?:tile|-icon)[-0-9x]*\.png|browserconfig.xml|apple-(?:touch-)?icon[-0-9x]*\.png|favicon[-0-9x]*.png|favicon\.ico|manifest.json|apple-touch-icon-precomposed\.png)$ {
        try_files $uri /static/favicon$uri @default;
    }   

    error_page 500 502 503 504 /500.html;
    location = /500.html {
        try_files /500.html /error.html /error500.html;
    }

    location / {
        try_files /maintenance.html @default;
    }
}

This config will simply pass request coming from any domain to socket located in /run/uwsgi/domain_name_without_www.sock. If it doesn't exist, nginx will throw some standard error. It will also try to serve static and media files directly from /sites/domain_name_without_www/public/ directory. If it fails, they will be served through uWSGI.

2. uWSGI emperor configuration

uWSGI have it's own multi-app deployment system, called Emperor. It can also start vassals (uWSGI instances) when particular socket is accessed. Example configuration for emperor may look like:

uwsgi --emperor /etc/uwsgi/apps-enabled --emperor-on-demand-directory /run/uwsgi --emperor-on-demand-extension .sock

That will load configs from /etc/uwsgi/apps-enabled, create socket for each config un /run/uwsgi and start uWSGI server for particular app only if nginx (or something else) is trying to access socket for that app. So it will behave almost exactly like your conception with symlink created in apps-enabled.

On next steps, you can configure each vassal to shut itself down when there are no requests from some time. Also you can create some pre-start and pre-stop hooks in vassals, so they can create symlink for nginx when vassal for app starts and remove them when vassal shuts down (with reloading nginx after each) so your nginx will route requests to that apps faster (it will be more efficient to route it using virtual host).