Store static files on S3 but staticfiles.json manifest locally

Question

I have a Django application running on Heroku. To store and serve my static files, I'm using django-storages with my S3 bucket, as well as the standard Django ManifestFilesMixin. I'm also using django-pipeline.

In code:

from django.contrib.staticfiles.storage import ManifestFilesMixin
from storages.backends.s3boto import S3BotoStorage
from pipeline.storage import PipelineMixin

class S3PipelineManifestStorage(PipelineMixin, ManifestFilesMixin, S3BotoStorage):
    pass

The setup works, however the staticfiles.json manifest is also stored on S3. I can see two problems with that:

1. My app's storage instance would have to fetch staticfiles.json from S3, instead of just getting it from the local file system. This makes little sense performance-wise. The only consumer of the manifest file is the server app itself, so it might as well be stored on the local file system instead of remotely.

   I'm not sure how significant this issue is since I suppose (or hope) that the server app caches the file after reading it once.

2. The manifest file is written during deployment by collectstatic, so if any already-running instances of the previous version of the server application read the manifest file from S3 before the deployment finishes and the new slug takes over, they could fetch the wrong static files - ones which should only be served for instances of the new slug.

   Note that specifically on Heroku, it's possible for new app instances to pop up dynamically, so even if the app does cache the manifest file, it's possible its first fetch of it would be during the deployment of the new slug.

   This scenario as described is specific to Heroku, but I guess there would be similar issues with other environments.

The obvious solution would be to store the manifest file on the local file system. Each slug would have its own manifest file, performance would be optimal, and there won't be any deployment races as described above.

Is it possible?

Answer 1

There is Django ticket #27590 that addresses this question. The ticket has a pull request that implements a solution, but it has not been reviewed yet.

Answer 2

The answer of @John and Kogan is great but doesn't give the full code needed to make this work: As @Danra mentioned you need to also save the staticfiles.json in the source folder to make this work. Here's the code I've created based on the above answer:

import json
import os
from django.conf import settings
from django.core.files.base import ContentFile
from django.core.files.storage import FileSystemStorage

from whitenoise.storage import CompressedManifestStaticFilesStorage
# or if you don't use WhiteNoiseMiddlware:
# from django.contrib.staticfiles.storage import ManifestStaticFilesStorage


class LocalManifestStaticFilesStorage(CompressedManifestStaticFilesStorage):
    """
    Saves and looks up staticfiles.json in Project directory
    """
    manifest_location = os.path.abspath(settings.BASE_DIR)  # or settings.PROJECT_ROOT depending on how you've set it up in your settings file.
    manifest_storage = FileSystemStorage(location=manifest_location)

    def read_manifest(self):

        try:
            with self.manifest_storage.open(self.manifest_name) as manifest:
                return manifest.read().decode('utf-8')
        except IOError:
            return None

    def save_manifest(self):
        payload = {'paths': self.hashed_files, 'version': self.manifest_version}
        if self.manifest_storage.exists(self.manifest_name):
            self.manifest_storage.delete(self.manifest_name)
        contents = json.dumps(payload).encode('utf-8')
        self.manifest_storage._save(self.manifest_name, ContentFile(contents))

Now you can use LocalManifestStaticFilesStorage for your STATICFILES_STORAGE. When running manage.py collectstatic, your manifest will be saved to your root project folder and Django will look for it there when serving the content.

If you have a deployment with multiple virtual machines, make sure to run collectstatic only once and copy the staticfiles.json file to all the machines in your deployment as part of your code deployment. The nice thing about this is that even if some machines don't have the latest update yet, they will still be serving the correct content (corresponding to the current version of the code), so you can perform a gradual deploy where there is a mixed state.

Answer 3

Some time ago I read this article which I believe fits your case well.
In there at the last paragraph exists the following:

Where is staticfiles.json located?

By default staticfiles.json will reside in STATIC_ROOT which is the directory where all static files are collected in.
We host all our static assets on an S3 bucket which means staticfiles.json by default would end up being synced to S3. However, we wanted it to live in the code directory so we could package it and ship it to each app server.

As a result of this, ManifestStaticFilesStorage will look for staticfiles.json in STATIC_ROOT in order to read the mappings. We had to overwrite this behaviour, so we subclassed ManifestStaticFilesStorage:

from django.contrib.staticfiles.storage import
ManifestStaticFilesStorage from django.conf import settings

class KoganManifestStaticFilesStorage(ManifestStaticFilesStorage):

    def read_manifest(self):
        """
        Looks up staticfiles.json in Project directory
        """
        manifest_location = os.path.abspath(
            os.path.join(settings.PROJECT_ROOT, self.manifest_name)
        )
        try:
            with open(manifest_location) as manifest:
                return manifest.read().decode('utf-8')
        except IOError:
            return None

With the above change, Django static template tag will now read the mappings from staticfiles.json that resides in project root directory.

Haven't used it myself, so let me know if it helps!