CODEWITHSUNDEEP
pythondjangoamazon-web-servicesamazon-ec2amazon-elb
rfj001
rfj001

Reputation: 8498

Django ALLOWED_HOSTS with ELB HealthCheck

I have a django application deployed on Elastic Beanstalk. The HealthCheck for my app keeps failing because the IP of the ELB HealthCheck is not included in my ALLOWED_HOSTS settings variable.

How can I modify ALLOWED_HOSTS to make the HealthCheck pass? I would just pass in the explicit IP address, but I believe that this changes, so whenever the IP changes the check would fail again until I add the new IP.

Upvotes: 22

Views: 12347

Answers (7)

Diogo Freire
Diogo Freire

Reputation: 11

If you're reading this in 2024 and still having issues even after implementing all the suggestions here, I just spent a ton of time trying to get to the bottom of this and managed to find a solution as my logs and Sentry were completely clogged.

Steps

  • Login to the EB EC2 instance.
  • Make a copy of the /etc/nginx/nginx.conf file. ABSOLUTELY DO THIS - I recommend putting that in your application code should the future require you to roll back.
  • Create a directive to replace calls from IP with a custom hostname that you can add to ALLOWED_HOSTS.
  • Copy the contents of /etc/nginx/conf.d/elasticbeanstalk/00_application.conf and put them in the server context of the above.
  • Update the proxy_set_header in the location directive to $my_host. Comment out or remove the directive include conf.d/elasticbeanstalk/*.conf; from your nginx.conf file.

What it looks like

user                    nginx;
error_log               /var/log/nginx/error.log warn;
pid                     /var/run/nginx.pid;
worker_processes        auto;
worker_rlimit_nofile    200000;

events {
    worker_connections  1024;
}

http {
    server_tokens off;

    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';

    include       conf.d/*.conf;

    map $http_upgrade $connection_upgrade {
        default     "upgrade";
    }

    server {
        listen        80 default_server;
        access_log    /var/log/nginx/access.log main;

        client_header_timeout 60;
        client_body_timeout   60;
        keepalive_timeout     60;
        gzip                  off;
        gzip_comp_level       4;
        gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript;

        # ELB MAGIC HERE - if we detect an IP, set host to a custom domain
        set $my_host $host;
        if ($host ~ "\d+\.\d+\.\d+\.\d+") {
            set $my_host "elb.mydomain.com"; # ADD THIS DOMAIN TO ALLOWED HOSTS
        }

        location / {
            proxy_pass          http://127.0.0.1:8000 ;
            proxy_http_version  1.1;

            proxy_set_header    Connection          $connection_upgrade;
            proxy_set_header    Upgrade             $http_upgrade;
            # THE BELOW WILL SET THE DOMAIN TO YOUR CUSTOM DOMAIN ABOVE IF IT'S FROM AN IP
            proxy_set_header    Host                $my_host;
            proxy_set_header    X-Real-IP           $remote_addr;
            proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
        } 

        location /static {
            alias /var/app/current/static;
            access_log off;
        }       

        # THE EXTRA FILES FROM ELB CONTAIN THE location / DIRECTIVE BELOW
        # P.S. I don't love this
        # Include the Elastic Beanstalk generated locations
        # include conf.d/elasticbeanstalk/*.conf;
    }
}

Full write up: https://diogofreire.btw.so/fixing-the-damn-allowed_hosts-issue-with-amazon-elastic-beanstalk-and-django

Upvotes: 1

Oscar Chen
Oscar Chen

Reputation: 636

I like Watt Imsuri's answer above, and I think I'm going to use this approach in all my projects so I turned it into a pip package for Django.

pip install django-easy-health-check

Add the health check middleware to Django settings before django.middleware.common.CommonMiddleware:

MIDDLEWARE = [
    ...,
    'easy_health_check.middleware.HealthCheckMiddleware',
    'django.middleware.common.CommonMiddleware',
    ...
]

By default, the health check url will be available at "example.com/healthcheck/".

You can also customize and overwrite the default settings by including the following in your project's settings.py:

DJANGO_EASY_HEALTH_CHECK = {
    "PATH": "/healthcheck/",
    "RETURN_STATUS_CODE": 200,
    "RETURN_BYTE_DATA": "",
    "RETURN_HEADERS": None
}

In production, you may also want to set the following Django settings:

ALLOWED_HOSTS = ["example.com"]
SECURE_SSL_REDIRECT = True
SECURE_REDIRECT_EXEMPT = [r'^healthcheck/$']

For more info, check out the git.

Upvotes: 1

Benas Volkovas
Benas Volkovas

Reputation: 61

Solution

The solution that worked for me was to simply install the django-ebhealthcheck library. After installing it, just add ebhealthcheck.apps.EBHealthCheckConfig to your INSTALLED_APPS.

From django-ebhealthcheck GitHub:

By default, Elastic Beanstalk's health check system uses the public IP of each load balanced instance as the request's host header when making a request. Unless added to ALLOWED_HOSTS, this causes Django to return a 400 Bad Request and a failed health check.

This app dynamically adds your instance's public IP address to Django's ALLOWED_HOSTS setting to permit health checks to succeed. This happens upon application start.

Version 2.0.0 and higher supports IMDSv2. If you are using v1 and cannot upgrade, use version 1 of this library instead (pip install django-ebhealthcheck<2.0.0).

Installation

  1. pip install django-ebhealthcheck

  2. Add ebhealthcheck.apps.EBHealthCheckConfig to your INSTALLED_APPS:

INSTALLED_APPS = [
    ...
    'ebhealthcheck.apps.EBHealthCheckConfig',
    ...
]

Upvotes: 6

sytech
sytech

Reputation: 41061

To expand on the answer provided by dfrdmn:

While this answer works well in most cases, it has a couple small potential problems.

AWS ELB Network Load Balancers

First, if you are using an ELB network load balancer, this method won't work with its HTTP health checks because the load balancer sends the IP address of the load balancer in the HTTP host header. From the AWS docs:

The HTTP host header in the health check request contains the IP address of the load balancer node and the listener port, not the IP address of the target and the health check port. If you are mapping incoming requests by host header, you must ensure that health checks match any HTTP host header. Another option is to add a separate HTTP service on a different port and configure the target group to use that port for health checks instead. Alternatively, consider using TCP health checks.

So, adding your instance (target group) IP to your ALLOWED_HOSTS will not work. As stated, you could use TCP health checks, or you could use the middleware approach described in another answer.

Metadata endpoint is throttled

Second, because the metadata endpoint limits number of concurrent connections and throttles requests, you may encounter issues in some cases.

Your Django settings.py file is executed for every process and any time processes need to restart. This is important if your webserver is configured to use multiple processes, such as when using gunicorn workers, as is commonly configured to properly take full advantage of system CPU resources.

This means that, with enough processes, your settings.py file will be executed many times, sending many concurrent requests to the metadata endpoint and your processes could fail to start. Further, on subsequent process restarts, the throttling will exacerbate the throttling problem. In some circumstances, this can cause your application to grind to a halt or have fewer processes running than intended.

To get around this, you could do a few things:

  1. Obtain the IP address before starting your server and set the IP address as an environment variable, then read the environment variable to add it to your allowed hosts.
$ export ALLOWED_HOST_EC2_PRIVATE_IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4)
$ gunicorn -w 10 ... myapp:app

# settings.py

ALLOWED_HOSTS = ['myhost.tld', ]

if os.getenv('ALLOWED_HOST_EC2_PRIVATE_IP'):
    ALLOWED_HOSTS.append(os.environ['ALLOWED_HOST_EC2_PRIVATE_IP'])

You may yet still encounter throttling issues with the metadata endpoint if many applications or other services utilize the instance's metadata at the same time.

  1. For services running in containers on ECS you can use the container metadata file

You can do this safely within the settings.py because there is no throttling or rate limit for accessing this file. This also avoids your application potentially interfering with other services that need the instance's metadata endpoint.

# settings.py
import os
import json

ALLOWED_HOSTS = ['myhost.tld', ]

if os.getenv('ECS_CONTAINER_METADATA_FILE'):
    metadata_file_path = os.environ['ECS_CONTAINER_METADATA_FILE']
    with open(metadata_file_path) as f:
        metadata = json.load(f)
    private_ip = metadata["HostPrivateIPv4Address"]
    ALLOWED_HOSTS.append(private_ip)

You could also combine the first approach with the metadata file, in your container's ENTRYPOINT.

#!/usr/bin/env bash
# docker-entrypoint.sh
export ALLOWED_HOST_EC2_PRIVATE_IP=$(jq -r .HostPrivateIPv4Address $ECS_CONTAINER_METADATA_FILE)

exec "$@"

FROM myapplication
COPY docker-entrypoint.sh /docker-entrypoint.sh
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD ["gunicorn", "whatever"]

Upvotes: 4

Watt Iamsuri
Watt Iamsuri

Reputation: 768

Here is another solution using Django Middleware.

Django's django.middleware.common.CommonMiddleware calls request.get_host(), which validates the request with ALLOWED_HOSTS. If you simply want to check that the application is running, you can create a middleware like this.

from django.http import HttpResponse


class HealthCheckMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        if request.path == '/health':
            return HttpResponse('ok')
        return self.get_response(request)

And put your HealthCheckMiddleware in front of CommonMiddleware in settings.py

MIDDLEWARE = [
    'yourdjangoapp.middleware.HealthCheckMiddleware',
    ......
    'django.middleware.common.CommonMiddleware',
    ......
]

And your application will always respond to path /health with ok as long as your app is running regardless of any configurations.

Upvotes: 40

KVISH
KVISH

Reputation: 13208

The other solution doesn't answer the question because it doesn't take into consideration all the various tools AWS has (ELB, etc). What we ended up doing (since we use nginx + uwsgi) is we set the header to something valid when the user sends a request.

As documented on this page:

https://opbeat.com/blog/posts/the-new-django-allowed_hosts-with-elb-uwsgi-and-nginx/

We put our nginx configuration as below:

set $my_host $host;
if ($host ~ "\d+\.\d+\.\d+\.\d+") {
  set $my_host "example.com";
}

location / {
  uwsgi_pass unix:///tmp/mysite.com.sock;
  uwsgi_param HTTP_HOST $my_host;
  include uwsgi_params;
}

The key here is to put a valid value for $my_host as per your ALLOWED_HOSTS.

Unfortunately there is no "perfect" solution without increasing overhead. Some configurations will require you to add IP addresses all the time to the ALLOWED_HOSTS but this solution would generally work with the least over head.

Upvotes: -1

dfrdmn
dfrdmn

Reputation: 580

Your EC2 instance can query metadata about itself, including its IP address which is available at: http://169.254.169.254/latest/meta-data/local-ipv4.

You can test this by ssh-ing into your EC2 instance and running:

curl http://169.254.169.254/latest/meta-data/local-ipv4

So, in your configuration, you can do something like:

import requests

ALLOWED_HOSTS = ['.yourdomain.com', ]
try:
    EC2_IP = requests.get('http://169.254.169.254/latest/meta-data/local-ipv4').text
    ALLOWED_HOSTS.append(EC2_IP)
except requests.exceptions.RequestException:
    pass

Obviously you can replace requests with urllib if you don't want the dependency.

Upvotes: 23

Related Questions