How to get Django to make some files public and media files private on AWS S3 (no 403 errors)?

Question

I'm using boto3 and django-storages in my Django app to serve files from AWS S3. I'd like my static files to be public but other files to be private. I've got it kinda sorta working but not quite. My static files are being served as if they're private, with a pre-signed key. In my template file when I use:

<img src="{% static 'images/3d-house-nav-gray.png' %}">

instead of what I want

<img src="https://mybucket.s3.amazonaws.com/static/images/3d-house-nav-gray.png">

I'm getting

<img id="home-img" src="https://mybucket.s3.amazonaws.com/static/images/3d-house-nav-gray.png?AWSAccessKeyId=AKIA1234564LQ7X4EGHK&amp;Signature=123456gIBTFlTQKCexLo3UJmoPs%3D&amp;Expires=1621693552">

This actually works when the templates are rendered from the server as part of an HTTPResponse, but not when an image like this is simply included as part of, say, a .css file. In that case I'll get:

Failed to load resource: the server responded with a status of 403 (Forbidden)

(I find that if I copy and paste the problematic image link and replace the & with an & then I have access, a further mystery.)

Here is how I have AWS configured:

AWS_ACCESS_KEY_ID = os.environ['AWS_ACCESS_KEY_ID']
AWS_SECRET_ACCESS_KEY = os.environ['AWS_SECRET_ACCESS_KEY']
AWS_STORAGE_BUCKET_NAME = 'mybucket'
AWS_S3_OBJECT_PARAMETERS = {
    'CacheControl': 'max-age=86400',
}
AWS_DEFAULT_ACL = None
AWS_LOCATION = 'static'
STATICFILES_STORAGE = 'myapp.storage_backends.StaticStorage'
DEFAULT_FILE_STORAGE = 'myapp.storage_backends.MediaStorage'
AWS_S3_URL = 'https://%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
STATIC_DIRECTORY = '/static/'
MEDIA_DIRECTORY = '/media/'
STATIC_URL = AWS_S3_URL + STATIC_DIRECTORY
MEDIA_URL = AWS_S3_URL + MEDIA_DIRECTORY

Where myapp.storage_backends.py contains:

from storages.backends.s3boto3 import S3Boto3Storage

class MediaStorage(S3Boto3Storage):
    location = 'media'
    file_overwrite = False

class StaticStorage(S3Boto3Storage):
    location = 'static'
    file_overwrite = True

And on AWS S3, my bucket policy is set up like so:

{
    "Version": "2012-10-17",
    "Id": "Policy1621539673651",
    "Statement": [
        {
            "Sid": "Stmt1621539665305",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::063896663644:user/mylogin"
            },
            "Action": [
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3:::mybucket/*"
        },
        {
            "Sid": "Stmt1621539600741",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::mybucket/static/*"
        }
    ]
}

How can I fix this to make certain files (like everything in static/) unsigned and public, but other files signed and private?

Answer 1

Late to the party, but I think it's because you didn't set AWS_S3_CUSTOM_DOMAIN.

The documentation doesn't really make this obvious IMO.

Evidence: The url() method in S3Boto3Storage class:

def url(self, name, parameters=None, expire=None, http_method=None):
    # Preserve the trailing slash after normalizing the path.
    name = self._normalize_name(clean_name(name))
    params = parameters.copy() if parameters else {}
    if expire is None:
        expire = self.querystring_expire

    if self.custom_domain:               # <------- !Important
        url = '{}//{}/{}{}'.format(
            self.url_protocol,
            self.custom_domain,
            filepath_to_uri(name),
            '?{}'.format(urlencode(params)) if params else '',
        )

        if self.querystring_auth and self.cloudfront_signer:
            expiration = datetime.utcnow() + timedelta(seconds=expire)
            return self.cloudfront_signer.generate_presigned_url(url, date_less_than=expiration)

        return url

    params['Bucket'] = self.bucket.name
    params['Key'] = name
    url = self.bucket.meta.client.generate_presigned_url('get_object', Params=params,
                                                         ExpiresIn=expire, HttpMethod=http_method)
    if self.querystring_auth:
        return url
    return self._strip_signing_parameters(url)

What you should notice is the if self.custom_domain part.

If the custom_domain attribute is set:

1. a url is created
2. if you're not using cloudfront, the URL is returned without the signed URL parameters.

Where does this self.custom_domain come from? S3Boto3Storage().get_default_settings()

...
'querystring_expire': setting('AWS_QUERYSTRING_EXPIRE', 3600),
'signature_version': setting('AWS_S3_SIGNATURE_VERSION'),
'location': setting('AWS_LOCATION', ''),
'custom_domain': setting('AWS_S3_CUSTOM_DOMAIN'),  # <-- looks for it in settings.py
'cloudfront_signer': cloudfront_signer,
'addressing_style': setting('AWS_S3_ADDRESSING_STYLE'),
...

Answer 2

If you take a look at the source code for django-storages, you will see an undocumented class called S3StaticStorage:

class S3StaticStorage(S3Boto3Storage):
    """Querystring auth must be disabled so that url() returns a consistent output."""
    querystring_auth = False

Subclassing from this should fix your problem.