Not getting static files with django & heroku deployment

Question

I'm developing a webapp with Django 1.7 (project_name is "sonata") and the project layout from the "Two scoops of Django 1.6", so I have a 3-tier basic folder tree.

.
├── docs
├── requirements
├── scripts
└── sonata
    ├── person
    │   └── templatetags
    ├── registration
    ├── sonata
    │   └── settings
    ├── static
    │   ├── css
    │   │   └── images
    │   ├── fonts
    │   └── js
    ├── templates
    │   ├── personApp
    │   └── registrationApp
    └── utils
        └── templatetags

I have a problem when deploying on Heroku. I have achieved the deployment, but the static files are not being served to the browser.

I know I should force some way of serving the static files through the settings and have googled about. I've seen many ways and read about using Amazon services, but I'm looking for the easiest one, which will make the future production deployment easy as well with gUnicorn (I hope).

I tried modifying the settings file (further down, is the heroku.py one, which overrides the base.py) and changing values for STATIC_ROOT, STATICFILES_DIRS and adding the line:

urlpatterns += static(settings.MEDIA_URL, document_root=settings.STATIC_ROOT)

I've tried as well running:

heroku run sonata/manage.py collectstatic

before doing

git push heroku master

but nothing happens, even checking with

heroku run ls sonata/assets 

that the files are being copied.

Please, I would like some orientation for really understanding what I'm doing wrong and mending it.

When finding out about heroku and deployment I met also some sample ProcFiles which used a project_name.wsgi file and I don't know anything about it.

I could use use some help, because the more webs I read, the more confused I get. Please assume I know very little about deployments. A link would be useful, but it has to show newbie's material :-(

Thank you very much on advance.

ProcFile:

web: python sonata/manage.py runserver 0.0.0.0:$PORT --noreload

heroku.py:

# -*- coding: utf-8 -*-
"""Heroku settings and globals."""

from __future__ import absolute_import

from .base import *

from os import environ

# TODO Warning! Heroku retrieve values as strings
# TODO we should check (only for Heroku) that 'True' and 'False' are respectively True and False 0, 
# or the equivalent ones, True and False
def get_env_setting(setting):
    """ Gets the environment variable or an Exception.
    This can be used, for example, for getting the SECRET_KEY and not having it hardcoded in the source code 
    Also for setting the active settings file for local development, heroku, production server, etc... """
    try:
        return environ[setting]
    except KeyError:
        error_msg = "Set the %s env variable" % setting
        raise ImproperlyConfigured(error_msg)

########## HOST CONFIGURATION
# See: https://docs.djangoproject.com/en/1.5/releases/1.5/#allowed-hosts-required-in-production
ALLOWED_HOSTS = ['*']
########## END HOST CONFIGURATION

########## EMAIL CONFIGURATION
# See: https://docs.djangoproject.com/en/dev/ref/settings/#email-backend
EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'

# See: https://docs.djangoproject.com/en/dev/ref/settings/#email-host
EMAIL_HOST = environ.get('EMAIL_HOST', 'smtp.gmail.com')

# See: https://docs.djangoproject.com/en/dev/ref/settings/#email-host-password
EMAIL_HOST_PASSWORD = environ.get('EMAIL_HOST_PASSWORD', '')

# See: https://docs.djangoproject.com/en/dev/ref/settings/#email-host-user
EMAIL_HOST_USER = environ.get('EMAIL_HOST_USER', 'your_email@example.com')

# See: https://docs.djangoproject.com/en/dev/ref/settings/#email-port
EMAIL_PORT = environ.get('EMAIL_PORT', 587)

# See: https://docs.djangoproject.com/en/dev/ref/settings/#email-subject-prefix
EMAIL_SUBJECT_PREFIX = '[%s] ' % SITE_NAME

# See: https://docs.djangoproject.com/en/dev/ref/settings/#email-use-tls
EMAIL_USE_TLS = True

# See: https://docs.djangoproject.com/en/dev/ref/settings/#server-email
SERVER_EMAIL = EMAIL_HOST_USER
########## END EMAIL CONFIGURATION

########## DATABASE CONFIGURATION
import dj_database_url
DATABASES['default'] = dj_database_url.config()
# DATABASES = {}
########## END DATABASE CONFIGURATION


########## CACHE CONFIGURATION
# See: https://docs.djangoproject.com/en/dev/ref/settings/#caches
CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
    }
}
########## END CACHE CONFIGURATION


########## SECRET CONFIGURATION
# See: https://docs.djangoproject.com/en/dev/ref/settings/#secret-key
SECRET_KEY = get_env_setting('SECRET_KEY')
########## END SECRET CONFIGURATION

base.py:

# -*- coding: utf-8 -*-
"""Common settings and globals."""

from os.path import abspath, basename, dirname, join, normpath
from sys import path


from os import environ

def get_env_setting(setting):
    """ Gets the environment variable or an Exception.
    This can be used, for example, for getting the SECRET_KEY and not having it hardcoded in the source code 
    Also for setting the active settings file for local development, heroku, production server, etc... """
    try:
        return environ[setting]
    except KeyError:
        error_msg = "Set the %s env variable" % setting
        raise ImproperlyConfigured(error_msg)


########## PATH CONFIGURATION
# Absolute filesystem path to the Django project directory:
DJANGO_ROOT = dirname(dirname(abspath(__file__)))

# Absolute filesystem path to the top-level project folder:
SITE_ROOT = dirname(DJANGO_ROOT)

# Site name:
SITE_NAME = basename(DJANGO_ROOT)

# Add our project to our pythonpath, this way we don't need to type our project
# name in our dotted import paths:
path.append(DJANGO_ROOT)
########## END PATH CONFIGURATION


########## DEBUG CONFIGURATION
# See: https://docs.djangoproject.com/en/dev/ref/settings/#debug
DEBUG = False

# See: https://docs.djangoproject.com/en/dev/ref/settings/#template-debug
TEMPLATE_DEBUG = DEBUG
########## END DEBUG CONFIGURATION


########## MANAGER CONFIGURATION
# See: https://docs.djangoproject.com/en/dev/ref/settings/#admins
ADMINS = (
    ('Your Name', 'your_email@example.com'),
)

# See: https://docs.djangoproject.com/en/dev/ref/settings/#managers
MANAGERS = ADMINS
########## END MANAGER CONFIGURATION


########## DATABASE CONFIGURATION
# See: https://docs.djangoproject.com/en/dev/ref/settings/#databases
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.',
        'NAME': '',
        'USER': '',
        'PASSWORD': '',
        'HOST': '',
        'PORT': '',
    }
}
########## END DATABASE CONFIGURATION


########## GENERAL CONFIGURATION
# See: https://docs.djangoproject.com/en/dev/ref/settings/#time-zone
TIME_ZONE = 'Europe/Madrid'

# See: https://docs.djangoproject.com/en/dev/ref/settings/#language-code
LANGUAGE_CODE = 'es-es'
DEFAULT_CHARSET = 'utf-8'

# See: https://docs.djangoproject.com/en/dev/ref/settings/#site-id
SITE_ID = 1

# See: https://docs.djangoproject.com/en/dev/ref/settings/#use-i18n
USE_I18N = True

LOCALE_PATHS = (# Idiomas disponibles en la aplicación
                SITE_ROOT + '/locale',
                )

LANGUAGES = (# Ruta donde buscar ficheros de idioma
                 ('es', 'Español'),
                 ('gl', 'Galego'),
                 ('en', 'English'),
                 ('it', 'Italiano'),
             )

# See: https://docs.djangoproject.com/en/dev/ref/settings/#use-l10n
USE_L10N = True

# See: https://docs.djangoproject.com/en/dev/ref/settings/#use-tz
USE_TZ = True
########## END GENERAL CONFIGURATION


########## MEDIA CONFIGURATION
# See: https://docs.djangoproject.com/en/dev/ref/settings/#media-root
MEDIA_ROOT = normpath(join(SITE_ROOT, 'media'))

# See: https://docs.djangoproject.com/en/dev/ref/settings/#media-url
MEDIA_URL = '/media/'
########## END MEDIA CONFIGURATION


########## STATIC FILE CONFIGURATION
# See: https://docs.djangoproject.com/en/dev/ref/settings/#static-root
STATIC_ROOT = normpath(join(SITE_ROOT, 'assets'))

# See: https://docs.djangoproject.com/en/dev/ref/settings/#static-url
STATIC_URL = '/static/'

# See: https://docs.djangoproject.com/en/dev/ref/contrib/staticfiles/#std:setting-STATICFILES_DIRS
STATICFILES_DIRS = (
    normpath(join(SITE_ROOT, 'static')),
)

# See: https://docs.djangoproject.com/en/dev/ref/contrib/staticfiles/#staticfiles-finders
STATICFILES_FINDERS = (
    'django.contrib.staticfiles.finders.FileSystemFinder',
    'django.contrib.staticfiles.finders.AppDirectoriesFinder',
)
########## END STATIC FILE CONFIGURATION

########## LOGIN REDIRECTION
# The URL you'd like to redirect users to that aren't logged in
LOGIN_URL = '/registration/login/'
#########################################

########## LOGIN NOT REQUIRED
# Tuple of regular expressions that lists your exceptions to the default login required on every page.
LOGIN_EXEMPT_URLS = (
 r'^registration/login\.html$',
 r'^admin/',
)
#########################################

########## SECRET CONFIGURATION
# See: https://docs.djangoproject.com/en/dev/ref/settings/#secret-key
# Note: This key should only be used for development and testing.
SECRET_KEY = r"I am not going to show you my secret key, sorry"
########## END SECRET CONFIGURATION


########## SITE CONFIGURATION
# Hosts/domain names that are valid for this site
# See https://docs.djangoproject.com/en/1.5/ref/settings/#allowed-hosts
ALLOWED_HOSTS = []
########## END SITE CONFIGURATION


########## FIXTURE CONFIGURATION
# See: https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-FIXTURE_DIRS
FIXTURE_DIRS = (
    normpath(join(SITE_ROOT, 'fixtures')),
)
########## END FIXTURE CONFIGURATION


########## TEMPLATE CONFIGURATION
# See: https://docs.djangoproject.com/en/dev/ref/settings/#template-context-processors
TEMPLATE_CONTEXT_PROCESSORS = (
    'django.contrib.auth.context_processors.auth',
    'django.core.context_processors.debug',
    'django.core.context_processors.i18n',
    'django.core.context_processors.media',
    'django.core.context_processors.static',
    'django.core.context_processors.tz',
    'django.contrib.messages.context_processors.messages',
    'django.core.context_processors.request',
)

# See: https://docs.djangoproject.com/en/dev/ref/settings/#template-loaders
TEMPLATE_LOADERS = (
    'django.template.loaders.filesystem.Loader',
    'django.template.loaders.app_directories.Loader',
)

# See: https://docs.djangoproject.com/en/dev/ref/settings/#template-dirs
TEMPLATE_DIRS = (
    normpath(join(SITE_ROOT, 'templates')),
    normpath(join(SITE_ROOT, 'templates/registration')),
    normpath(join(SITE_ROOT, 'templates/person')),
)
########## END TEMPLATE CONFIGURATION


########## MIDDLEWARE CONFIGURATION
# See: https://docs.djangoproject.com/en/dev/ref/settings/#middleware-classes
MIDDLEWARE_CLASSES = (
    # Default Django middleware.
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.locale.LocaleMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    # Custom middleware
    'sonata.settings.middleware.LoginRequiredMiddleware',
)
########## END MIDDLEWARE CONFIGURATION


########## URL CONFIGURATION
# See: https://docs.djangoproject.com/en/dev/ref/settings/#root-urlconf
ROOT_URLCONF = '%s.urls' % SITE_NAME
########## END URL CONFIGURATION


########## APP CONFIGURATION
DJANGO_APPS = (
    # Default Django apps:
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'django.contrib.messages',
    'django.contrib.staticfiles',

    # Useful template tags:
    # 'django.contrib.humanize',

    # Admin panel and documentation:
    'django.contrib.admin',
    # 'django.contrib.admindocs',
)

# Apps specific for this project go here.
LOCAL_APPS = (
    'person',
    'registration',
    'utils',
)

# See: https://docs.djangoproject.com/en/dev/ref/settings/#installed-apps
INSTALLED_APPS = DJANGO_APPS + LOCAL_APPS
########## END APP CONFIGURATION


########## LOGGING CONFIGURATION
# See: https://docs.djangoproject.com/en/dev/ref/settings/#logging
# A sample logging configuration. The only tangible logging
# performed by this configuration is to send an email to
# the site admins on every HTTP 500 error when DEBUG=False.
# See http://docs.djangoproject.com/en/dev/topics/logging for
# more details on how to customize your logging configuration.
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'filters': {
        'require_debug_false': {
            '()': 'django.utils.log.RequireDebugFalse'
        }
    },
    'handlers': {
        'mail_admins': {
            'level': 'ERROR',
            'filters': ['require_debug_false'],
            'class': 'django.utils.log.AdminEmailHandler'
        }
    },
    'loggers': {
        'django.request': {
            'handlers': ['mail_admins'],
            'level': 'ERROR',
            'propagate': True,
        },
    }
}
########## END LOGGING CONFIGURATION


########## WSGI CONFIGURATION
# See: https://docs.djangoproject.com/en/dev/ref/settings/#wsgi-application
WSGI_APPLICATION = '%s.wsgi.application' % SITE_NAME

Answer 1

I finally got it. I had to include the following lines in heroku.py:

heroku.py

# Important for Heroku
BASE_DIR = dirname(dirname(abspath(__file__)))

STATIC_ROOT = 'static' # Important for Heroku
STATIC_URL = '/static/'

STATICFILES_DIRS = (
    path.join(BASE_DIR, 'static'),  # Important for Heroku
)

I had to add some lines to wsgi.py for static files:

wsgi.py

from django.core.wsgi import get_wsgi_application
from whitenoise.django import DjangoWhiteNoise

application = get_wsgi_application()
application = DjangoWhiteNoise(application)

and also change the Procfile:

Procfile

web: gunicorn --chdir sonata --log-file - sonata.wsgi:application

Answer 2

I'm developing a webapp with Django 1.7 (project_name is "sonata") and the project layout from the "Two scoops of Django 1.6", so I have a 3-tier basic folder tree.

The Two Scoops template that you are referring to is only for 1.6, you should not use it for 1.7.

For django 1.7, you should use the cookiecutter recipe (this is a link to the version for 1.7; the default repository is for django 1.8.)

I do not know why you have a heroku.py file there (this is not part of the template from 2scoops) and is not needed.

The default template from 2scoops assumes the following:

1. You will be using Heroku for deploying your application code.
2. You will be using AWS S3 for deploying your static files.
3. You will be using the default (free) postgresql service from Heroku for your database.
4. You will be using sendgrid (again, from Heroku) to send emails.
5. You will use memcachier (a hosted memcache service, again from Heroku) for your cache.

Make sure that is how you want to get started, because you'll need to setup AWS credentials separately.

Assuming you have started from scratch with the correct template for django 1.7, to launch it on heroku you would have to make sure you set the appropriate environment variables for Heroku, to make sure things can connect appropriately. The steps to do this are listed in the default readme (which is generated as part of your project).

Please make sure you follow the guidelines to make sure your project is setup correctly.

As a beginner, this may be a lot to digest especially if you are not familiar with django/heroku/AWS.

I suggest instead you follow the heroku tutorial for django which is a lot simpler and should get you started immediately.

Answer 3

The Procfile tells Heroku what processes it should start. Without it, Heroku has no way of starting the application. But first of all, for a Python application you need a requirements.txt file at the root of your repository.

I'm not familiar with Two Scoops of Django (I know about the book, but I've never read it), so I'm not sure what that project layout is all about. It's not wrong for sure, but it seems overly complicated to me, especially for a total beginner since it deviates somewhat from the “canonical” way of laying out a Django project. So I would really recommend that if you're just starting out with Django to take a look and work through the official Django tutorial (the First Steps section on that page).

Now assuming your current layout, for Heroku you need two files in the root of your repository (at the same level where the docs, scripts etc. directories are): Procfile and requirements.txt. Assuming your WSGI entry point is in the sonata/sonata directory (it should be called wsgi.py), Procfile should have something like this:

web: gunicorn sonata.sonata.wsgi --log-file -

Note that the Procfile you presented here should be able to run (assuming Python path is correct), but you're using the development server which should not be done in deployment environment. That gunicorn thing in my Procfile is a standard production quality Python WSGI application server and the Procfile simply says run gunicorn with this application.

The requirements.txt file should have at least:

Django>=1.7
gunicorn

in it. This file tells Heroku which Python packages to install with Pip (Python's package manager). In this case the latest version of Django 1.7 branch and Gunicorn. It should also list other Python dependencies your project has. The presence of this file is also what tells Heroku your application is a Python application. You should really carefully read the Getting started with Django on Heroku guide as it has the basics covered and pointers to further reading. I agree that Heroku's documentation is a bit all over the place for a total beginner, but if you take one step at a time and use Google for things that are unclear to you, you should have a working setup relatively quickly.

As for static files, yes, you can do the thing with the static() function in urlpatterns even if you're using Gunicorn instead of the development server but only if DEBUG = True in the settings.py, which it should absolutely not be in production! If this is just a staging or a test, it's OK, but even then the fundamental fact remains: Heroku is just an application server basically. It is not meant for serving of any static files. You can theoretically serve them from Heroku with the static() trick but as soon as you do a new deployment or as soon as the app restarts (which can and does happen frequently for a number of reasons) all of your static files will be gone. So, practically, you have to host your static files somewhere else, be it S3 or something other.

You also have a possible mistake in how you set up the static() function, it should be (if you also want to serve media files):

urlpatterns += static(settings.STATIC_URL, document_root=settings.STATIC_ROOT) +\
               static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

Again, this will work only for a short time until your app restarts for whatever reason, at which point you'll have a broken app because all the static files are gone.

Also note that you do not need to run collectstatic manually on Heroku. It will detect a Django application and run it automatically on deploy. Even if it didn't, you can't do it before you push your repository. It's the other way around, you first push and then collectstatic runs (you'll see it being run automatically).

It seems to me you really have your head tied in a knot with this, so my advice would be to take a step back. Make a very simple vanilla Django app (by following the tutorial I mentioned), and try deploying that to Heroku for which you should follow their Django deployment basics I linked. Even with that approach there are possible caveats, but those are out of the scope of this question. One step at a time (go as far back as you need until you figure it out) and you'll be fine.