django + nginx in docker containers: can not upload any file with forms

Question

I am trying to migrate my app to docker containers, but met the problem uploading files with django forms. Everything works just fine, but when I try to upload any file from forms in my app I get nothing, like multipart/form-data is not set in html form tag. But it set and everythings works if I install application directly with no dockers. Anyways here is my configs, I hope somebody can help me.

Here is my docker-compose.yml

version: '2' 
services:
    db_postgres:
        build:
            context: .
            dockerfile: dockerfiles/docker-postgres/Dockerfile
            args:
            - db_user=username
            - db_name=databasename
            - db_pass=password
        environment:
            LC_ALL: C.UTF-8

    app:
        restart: always
        build:
            context: .
            dockerfile: dockerfiles/docker-app/Dockerfile
        links:
            - db_postgres:db_postgres

    nginx:
        restart: always
        build: dockerfiles/docker-nginx
        volumes_from:
            - app
        ports:
            - "80:80"
            - "443:443"
        links:
            - app:applink

Here is application Dockerfile:

FROM ubuntu:16.04

RUN \
  apt-get update && \
  apt-get install -y python-pip python-dev build-essential python-virtualenv && \
  apt-get install -y libjpeg8 libjpeg62-dev libfreetype6 libfreetype6-dev && \
  apt-get install -y libpq-dev libffi-dev && \
  apt-get install -y libssl-dev git 

RUN mkdir app
COPY requrements.txt /app
RUN pip install --upgrade pip
RUN pip install -r /app/requrements.txt
ADD . /app
WORKDIR /app
VOLUME ["/app/staticfiles/", "/app/media/", "/app/protected/"]
# I tried this but it seams no effect at all
# this is media folders where users can upload their files
# nginx can read from this folders with no problem
# I tried to docker exec and nginx can even write there
# anyways I tried to start nginx as root
RUN chown www-data:www-data -R /app/media/
RUN chown www-data:www-data -R /app/protected/    

ADD dockerfiles/docker-app/django_entrypoint.sh .
RUN chmod +x django_entrypoint.sh  
CMD ["./django_entrypoint.sh"]
EXPOSE 8000

Here is django_entrypoint.sh

#!/bin/bash
NAME=my_app_name
USER=www-data
GROUP=www-data
NUM_WORKERS=8
DJANGO_WSGI_MODULE=my_application.wsgi

python manage.py makemigrations
python manage.py migrate
echo "Collenting staticfiles..."
python manage.py collectstatic --noinput > /dev/null
python manage.py initadmin
python manage.py init_default_settings
exec gunicorn ${DJANGO_WSGI_MODULE}:application \
  --name $NAME \
  --workers $NUM_WORKERS \
  --user=$USER --group=$GROUP \
  --bind=:8000 \
  --log-level=debug \
  --capture-output

here is nginx Dockerfile:

FROM ubuntu:16.04
# Install Nginx.
RUN apt-get update && \
  apt-get install -y nginx && \
  rm -rf /var/lib/apt/lists/* && \
  echo "
daemon off;" >> /etc/nginx/nginx.conf && \
  chown -R www-data:www-data /var/lib/nginx

# Define mountable directories.
VOLUME ["/etc/nginx/sites-available", "/etc/nginx/certs", "/etc/nginx/conf.d"]
ADD confgfile /etc/nginx/sites-available/
RUN rm /etc/nginx/sites-enabled/default && rm /etc/nginx/sites-available/default
RUN ln -s /etc/nginx/sites-available/ctrd /etc/nginx/sites-enabled/ctrd
# Define default command.
CMD ["nginx"]
# Expose ports.
EXPOSE 80
EXPOSE 443

Here is configfile for nginx:

server {
    listen 80;
    # I pasted my server ip in sever name
    server_name 175.116.110.231;
    client_max_body_size 300M;

    error_log stdout debug;
    location /static/ {
        alias   /app/staticfiles/;
    }
    location /media/ {
        alias   /app/media/;
    }
    location / {
        try_files $uri @proxy;
    }

    location @proxy {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-FILE $request_body_file;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass http://applink:8000;
    }

    location /private-uploads/ {
        internal;
        alias /app/protected/records/;
    }

}

The app works fine, but when I want to upload any file it does nothing, no file, no errors in errorlogs. If filefield can not be empty (requred=True in django forms), I got the error about that. In request in broswer tools:

Request URL:http://175.116.110.231/edit-avatar/2
Request Method:POST
Status Code:302 Found
Remote Address:175.116.110.231:80
Response Headers
view source
Connection:keep-alive
Content-Language:ru
Content-Type:text/html; charset=utf-8
Date:Mon, 19 Sep 2016 13:26:59 GMT
Location:/profile/
Server:nginx/1.10.0 (Ubuntu)
Transfer-Encoding:chunked
Vary:Accept-Language, Cookie
X-Frame-Options:SAMEORIGIN
Request Headers
view source
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip, deflate
Accept-Language:ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4
Cache-Control:max-age=0
Connection:keep-alive
Content-Length:30316
Content-Type:multipart/form-data; boundary=----WebKitFormBoundarySCXVqlqPsCHyaHtU
Cookie:JSESSIONID=dummy; sessionid=htnbd9coal4ws2ansxzze8bhtu4fq6do; csrftoken=tCD5cVrR0IUGkjkkbJKDsxdRrtyLIUGbOIHkjHKjhkjhmnVJhvKUGkBDjZ
DNT:1
Host:175.116.110.231
Origin:http://175.116.110.231
Referer:http://175.116.110.231/edit-avatar/2
Upgrade-Insecure-Requests:1
User-Agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.90 Safari/537.36 Vivaldi/1.4.589.11
Request Payload
------WebKitFormBoundarySCXVqlqPsCHyaHtU
Content-Disposition: form-data; name="csrfmiddlewaretoken"

zaAvb1KoCCovbuCbik261UDeZgQbJdumzcvpcqOHqTKIrRN826lEoeb5AvU7SrG6
------WebKitFormBoundarySCXVqlqPsCHyaHtU
Content-Disposition: form-data; name="avatar"; filename="ava2.jpg"
Content-Type: image/jpeg


------WebKitFormBoundarySCXVqlqPsCHyaHtU--

Upd gunicorn debug output at startup:

ctrd_app_1      | [2016-09-19 16:33:47 +0000] [1] [DEBUG] Current configuration:
ctrd_app_1      |   secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
ctrd_app_1      |   proxy_protocol: False
ctrd_app_1      |   worker_connections: 1000
ctrd_app_1      |   statsd_host: None
ctrd_app_1      |   max_requests_jitter: 0
ctrd_app_1      |   post_fork: 
ctrd_app_1      |   pythonpath: None
ctrd_app_1      |   enable_stdio_inheritance: False
ctrd_app_1      |   worker_class: sync
ctrd_app_1      |   ssl_version: 3
ctrd_app_1      |   suppress_ragged_eofs: True
ctrd_app_1      |   syslog: False
ctrd_app_1      |   syslog_facility: user
ctrd_app_1      |   when_ready: 
ctrd_app_1      |   pre_fork: 
ctrd_app_1      |   cert_reqs: 0
ctrd_app_1      |   preload_app: False
ctrd_app_1      |   keepalive: 2
ctrd_app_1      |   accesslog: None
ctrd_app_1      |   group: 33
ctrd_app_1      |   graceful_timeout: 30
ctrd_app_1      |   do_handshake_on_connect: False
ctrd_app_1      |   spew: False
ctrd_app_1      |   workers: 8
ctrd_app_1      |   proc_name: django_ctrd_app
ctrd_app_1      |   sendfile: None
ctrd_app_1      |   pidfile: None
ctrd_app_1      |   umask: 0
ctrd_app_1      |   on_reload: 
ctrd_app_1      |   pre_exec: 
ctrd_app_1      |   worker_tmp_dir: None
ctrd_app_1      |   post_worker_init: 
ctrd_app_1      |   limit_request_fields: 100
ctrd_app_1      |   on_exit: 
ctrd_app_1      |   config: None
ctrd_app_1      |   logconfig: None
ctrd_app_1      |   check_config: False
ctrd_app_1      |   statsd_prefix: 
ctrd_app_1      |   proxy_allow_ips: ['127.0.0.1']
ctrd_app_1      |   pre_request: 
ctrd_app_1      |   post_request: 
ctrd_app_1      |   user: 33
ctrd_app_1      |   forwarded_allow_ips: ['127.0.0.1']
ctrd_app_1      |   worker_int: 
ctrd_app_1      |   threads: 1
ctrd_app_1      |   max_requests: 0
ctrd_app_1      |   limit_request_line: 4094
ctrd_app_1      |   access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
ctrd_app_1      |   certfile: None
ctrd_app_1      |   worker_exit: 
ctrd_app_1      |   chdir: /app
ctrd_app_1      |   paste: None
ctrd_app_1      |   default_proc_name: calltrade.wsgi:application
ctrd_app_1      |   errorlog: -
ctrd_app_1      |   loglevel: debug
ctrd_app_1      |   capture_output: True
ctrd_app_1      |   syslog_addr: udp://localhost:514
ctrd_app_1      |   syslog_prefix: None
ctrd_app_1      |   daemon: False
ctrd_app_1      |   ciphers: TLSv1
ctrd_app_1      |   on_starting: 
ctrd_app_1      |   worker_abort: 
ctrd_app_1      |   bind: [':8000']
ctrd_app_1      |   raw_env: []
ctrd_app_1      |   reload: False
ctrd_app_1      |   limit_request_field_size: 8190
ctrd_app_1      |   nworkers_changed: 
ctrd_app_1      |   timeout: 30
ctrd_app_1      |   ca_certs: None
ctrd_app_1      |   django_settings: None
ctrd_app_1      |   tmp_upload_dir: None
ctrd_app_1      |   keyfile: None
ctrd_app_1      |   backlog: 2048
ctrd_app_1      |   logger_class: gunicorn.glogging.Logger
ctrd_app_1      | [2016-09-19 16:33:47 +0000] [1] [INFO] Starting gunicorn 19.6.0
ctrd_app_1      | [2016-09-19 16:33:47 +0000] [1] [DEBUG] Arbiter booted
ctrd_app_1      | [2016-09-19 16:33:47 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
ctrd_app_1      | [2016-09-19 16:33:47 +0000] [1] [INFO] Using worker: sync
ctrd_app_1      | [2016-09-19 16:33:47 +0000] [38] [INFO] Booting worker with pid: 38
ctrd_app_1      | [2016-09-19 16:33:47 +0000] [39] [INFO] Booting worker with pid: 39
ctrd_app_1      | [2016-09-19 16:33:47 +0000] [44] [INFO] Booting worker with pid: 44
ctrd_app_1      | [2016-09-19 16:33:47 +0000] [47] [INFO] Booting worker with pid: 47
ctrd_app_1      | [2016-09-19 16:33:47 +0000] [48] [INFO] Booting worker with pid: 48
ctrd_app_1      | [2016-09-19 16:33:47 +0000] [51] [INFO] Booting worker with pid: 51
ctrd_app_1      | [2016-09-19 16:33:47 +0000] [54] [INFO] Booting worker with pid: 54
ctrd_app_1      | [2016-09-19 16:33:48 +0000] [59] [INFO] Booting worker with pid: 59
ctrd_app_1      | [2016-09-19 16:33:48 +0000] [1] [DEBUG] 8 workers

I also tried 8000:8000 port expose on gunicorn container and It also can not save file, so the reason is not nginx, but my bad configuration of docker volumes maybe. Can someone explain docker volumes? PLease check my config of volumes, I must be do not understand how it should work. Please help.

Here is media serving view:

@login_required
def private_media_response(request, username, filename):
    """
    In the nginx setting we can use something like here:
    location /private-uploads/ {
        internal;
        alias /place/to/private/media/;
    }
    """
    user = request.user
    if user.username == username:
        response = HttpResponse()
        url = '/private-uploads/{0}/{1}'.format(username, filename)
        response.status_code = 200
        response['X-Accel-Redirect'] = url.encode('utf-8')
        response['X-Accel-Buffering'] = 'yes'
        return response
    else:
        return HttpResponseForbidden("Restricted Access")

The problem is not to get the file, I can not save it with forms. All my views that saving files looks like model update CBV or this standart view:

@login_requred
def file_save_view(request, **kwags):
    if request.POST:
        form = MyForm(request.POST, request.FILES)
        if form.is_valid()
            form.save()
    else:
        form = myForm();
    return render(request, 'some_template.html', {'form':form})

Standart models save file data in media, for protected files I create FileSystemStorage

location = os.path.join(settings.BASE_DIR, 'protected')
fs = FileSystemStorage(location=location)

and then use it in models:

def get_upload_path(instance, filename):
    user = instance.user
    return 'documents/{0}/{1}'.format(user, filename)

class MyModel(models.Model):
    date = models.DateTimeField(auto_now=True, blank=False)
    user = models.ForeignKey(User)
    document = models.FileField(
        upload_to=get_upload_path,
        storage=fs,
    )
    pass

This code little bit abstract, but it works on standart deploy way. Must be something wrong with volumes I guess.

Paul Becotte · Accepted Answer

So, you do seem to misunderstand volumes to some extent (though I don't see anything that would cause your problem).

Volumes come from the "copy on write" filesystem that Docker uses for its images. Each layer only contains new information that hasn't been changed since previous uses- this lets you build images in an efficient fashion without having to copy tons of data around for each container you start off of that image.

A volume is saying "do not use the copy on write filesystem for this directory". So VOLUME ["/etc/nginx/sites-available", "/etc/nginx/certs", "/etc/nginx/conf.d"] is actually telling Docker to leave those directories OUT of the image you build. This is not what you want... I see you adding config files to your nginx configuration AFTER marking the config directory as a volume... those files changes will not propogate into the image you are building.

On to your actual problem though- I have no idea what that is. None of the deployment configuration has anything to do with "saving a file". The only place it could is if you were trying to save it from django and then serve it back up with nginx... but you said the file isn't getting saved, which means that nginx has nothing to do with it. I would add some debug output to your app itself to try and figure out what is going on. If it is returning a 200 without doing what it says it is (saving some file to disk)... that is where I would start.

django + nginx in docker containers: can not upload any file with forms

Answers (2)

Related Questions