Django collectstatic not working on production with S3, but same settings work locally

Question

I've been moving around some settings to make more defined local and production environments, and I must have messed something up.

Below are the majority of relevant settings. If I move the production.py settings (which just contains AWS-related settings at the moment) to base.py, I can update S3 from my local machine just fine. Similarly, if I keep those AWS settings in base.py and push to production, S3 updates appropriately. In addition, if I print something from production.py, it does print. However, if I make production.py my "local" settings on manage.py, or when I push to Heroku with the settings as seen below, S3 is not updating.

What about my settings is incorrect? (Well, I'm sure a few things, but specifically causing S3 not to update?)

Here's some relevant code:

__init__.py (in the directory with base, local, and production)

from cobev.settings.base import *

base.py

INSTALLED_APPS = [
    ...
    'whitenoise.runserver_nostatic',
    'django.contrib.staticfiles',
    ...
    'storages',
]

...

STATIC_URL = '/static/'

STATICFILES_DIRS = [os.path.join(BASE_DIR, "global_static"),
                    os.path.join(BASE_DIR, "media", )
                    ]

MEDIA_ROOT = os.path.join(BASE_DIR, 'media/')
MEDIA_URL = '/media/'

local.py

# local_settings.py
from .base import *

...

STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')

production.py

from .base import *

# AWS Settings

AWS_ACCESS_KEY_ID = config('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = config('AWS_SECRET_ACCESS_KEY')
AWS_STORAGE_BUCKET_NAME = 'cobev'

AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
AWS_S3_OBJECT_PARAMETERS = {
    'CacheControl': 'max-age=86400',
}
AWS_LOCATION = 'static'

AWS_DEFAULT_ACL = 'public-read'

STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
DEFAULT_FILE_STORAGE = 'cobev.storage_backends.MediaStorage'
STATIC_URL = 'https://%s/%s/' % (AWS_S3_CUSTOM_DOMAIN, AWS_LOCATION)

ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'

# End AWS

wsgi.py

import os

from django.core.wsgi import get_wsgi_application

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "cobev.settings.production")

application = get_wsgi_application()

from whitenoise.django import DjangoWhiteNoise
application = DjangoWhiteNoise(application)

manage.py

#!/usr/bin/env python
import os
import sys

if __name__ == "__main__":
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "cobev.settings.local") 
    try:
        from django.core.management import execute_from_command_line
    except ImportError as exc:
        raise ImportError(
            "Couldn't import Django. Are you sure it's installed and "
            "available on your PYTHONPATH environment variable? Did you "
            "forget to activate a virtual environment?"
        ) from exc
    execute_from_command_line(sys.argv)

Answer 1

After a couple of hours searching for a solution. Just add these in the settings.py

They simply work

from django.conf import settings
from storages.backends.s3boto3 import S3Boto3Storage

class StaticStorage(S3Boto3Storage):
    location = settings.STATICFILES_LOCATION
    querystring_auth = False

class MediaStorage(S3Boto3Storage):
    location = settings.MEDIAFILES_LOCATION
    file_overwrite = False

STORAGES = {
    "default": {"BACKEND": "myfitness.custom_storage.MediaStorage"},
    "staticfiles": {"BACKEND": "myfitness.custom_storage.StaticStorage"},
}
    # Static files (CSS, JavaScript, etc.)
STATICFILES_LOCATION = 'static/'
STATIC_URL = f"https://{AWS_S3_CUSTOM_DOMAIN}/{STATICFILES_LOCATION}/"

MEDIAFILES_LOCATION = "media"
MEDIA_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/{MEDIAFILES_LOCATION}/'

Answer 2

It looks like you are using Whitenoise. Whitenoise allows django to serve its static files. It is different approach if you want to serve them from AWS.

So, you need to remove Whitenoise in order to utilize django-storages. Remove it from settings, middleware, wsgi.py, etc.

Also, you can remove everything from __init__.py in settings - settings file to be used is set by DJANGO_SETTINGS_MODULE env variable.

---

According to your STATICFILES_DIRS, media directory is included as static files. It is preferred for media to be served separately from static files (one of differences - static files are more likely to be cached and gziped) - i.e. also with AWS but from separate S3 bucket.

You may also add AWS CloudFront as CDN in front of your buckets later.

Answer 3

Then if that can help here is my config for AWS/Django/S3 in production :

Common Static config :

# STATIC FILE CONFIGURATION
# ------------------------------------------------------------------------------
# See: https://docs.djangoproject.com/en/dev/ref/settings/#static-root
STATIC_ROOT = str(ROOT_DIR('staticfiles'))

# See: https://docs.djangoproject.com/en/dev/ref/settings/#static-url
STATIC_URL = '/static/'

# See: https://docs.djangoproject.com/en/dev/ref/contrib/staticfiles/#std:setting-STATICFILES_DIRS
STATICFILES_DIRS = [
    str(APPS_DIR.path('static'))
]

# See: https://docs.djangoproject.com/en/dev/ref/contrib/staticfiles/#staticfiles-finders
STATICFILES_FINDERS = [
    'django.contrib.staticfiles.finders.FileSystemFinder',
    'django.contrib.staticfiles.finders.AppDirectoriesFinder'
]

Common Media config :

# MEDIA CONFIGURATION
# ------------------------------------------------------------------------------

# See: https://docs.djangoproject.com/en/dev/ref/settings/#media-root
MEDIA_ROOT = str(APPS_DIR('media'))

# See: https://docs.djangoproject.com/en/dev/ref/settings/#media-url
MEDIA_URL = '/media/'

Production Static config :

# STATIC CONFIG PRODUCTION
# ------------------------------------------------------------------------------
# See: http://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html

AWS_STORAGE_BUCKET_NAME = 'mybucket-name-production'
AWS_ACCESS_KEY_ID = 'YOUR_KEY_ID'
AWS_SECRET_ACCESS_KEY = 'YOUR_SECRET_KEY'
AWS_S3_HOST = "s3.amazonaws.com"
AWS_S3_URL = 'https://{bucker_name}.s3.amazonaws.com/'.format(bucker_name=AWS_STORAGE_BUCKET_NAME)

AWS_LOCATION = 'static/'

AWS_S3_URL_PROTOCOL = 'https:'
AWS_S3_CUSTOM_DOMAIN = 'static.mydomain.com' # I use sub domaine to serve static 

STATIC_URL = '{AWS_S3_URL_PROTOCOL}//{AWS_S3_CUSTOM_DOMAIN}/{AWS_LOCATION}'.format(
    AWS_S3_URL_PROTOCOL=AWS_S3_URL_PROTOCOL,
    AWS_S3_CUSTOM_DOMAIN=AWS_S3_CUSTOM_DOMAIN,
    AWS_LOCATION=AWS_LOCATION)

AWS_QUERYSTRING_AUTH = False

AWS_IS_GZIPPED = True
AWS_EXPIREY = 60 * 60 * 24 * 14

# For s3boto
AWS_HEADERS = {
    'Cache-Control': 'max-age=%d, s-maxage=%d, must-revalidate' % (AWS_EXPIREY, AWS_EXPIREY)
}

# For s3boto3
AWS_S3_OBJECT_PARAMETERS = {
    'CacheControl': 'max-age=%d' % AWS_EXPIREY,
}

AWS_PRELOAD_METADATA = True
#AWS_S3_FILE_OVERWRITE = True

STATICFILES_STORAGE = 'config.storages.StaticStorage'
DEFAULT_FILE_STORAGE = 'config.storages.DefaultStorage'



# MEDIA S3 CONFIG PRODUCTION
# --------------------------------------------------------------------------------

AWS_MEDIA_DIR = 'media'
MEDIA_URL = AWS_S3_URL + AWS_MEDIA_DIR + '/'
MEDIA_ROOT = MEDIA_URL

Here is my StaticStorage Class :

from storages.backends.s3boto3 import S3Boto3Storage


class StaticStorage(S3Boto3Storage):
    location = 'static'
    file_overwrite = False


class DefaultStorage(S3Boto3Storage):
    location = ''
    file_overwrite = False

After That I add command in the folder .ebextensions in .config file for collectstatic :

# ./ebextensions/02_container_commands.config file : 

container_commands:
  0.3.0.push.static.to.s3:
    command: "source /opt/python/run/venv/bin/activate && python manage.py collectstatic --ignore=.scss --noinput"
    leader_only: true
    ignoreErrors: true

Answer 4

Ok, let me try, as discovered in comments from the question, you do S3 update using collectstatic, but this is a management command which is called using manage.py file where you set cobev.settings.local as settings which are not equal to cobev.settings.production which is used for wsgi.py file.

I think you should manage your settings file using, normal Django way, OS environment variable named DJANGO_SETTINGS_MODULE.

For sure you should be able to set it in any production environment you are running.