botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

Question

I'm using django-dbbackup to back up my postgresql database to my s3 bucket. It's connected to my S3 bucket via the following settings:

draft1.settings.py

DBBACKUP_STORAGE = 'draft1.aws.utils.BackupRootS3BotoStorage'
DBBACKUP_S3_BUCKET = AWS_STORAGE_BUCKET_NAME
DBBACKUP_S3_ACCESS_KEY = AWS_ACCESS_KEY_ID
DBBACKUP_S3_SECRET_KEY = AWS_SECRET_ACCESS_KEY

draft1.aws.utils

BackupRootS3BotoStorage  = lambda: S3Boto3Storage(location='backup')

Bucket policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Allow All",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::****-bucket/*"
        },
        {
            "Sid": "Deny All Actions On All But Media and Static Unless Defined User",
            "Effect": "Deny",
            "NotPrincipal": {
                "AWS": "arn:aws:iam::********:root"
            },
            "Action": "s3:*",
            "NotResource": [
                "arn:aws:s3:::****-bucket/media/*",
                "arn:aws:s3:::****-bucket/static/*",
                "arn:aws:s3:::****-bucket/media_thumbnail/*"
            ]
        }
    ]
}

As you can see I'm trying to back it up to the backup folder.

Full error:

Traceback (most recent call last):
  File "manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/home/zorgan/postr/env/lib/python3.5/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/home/zorgan/postr/env/lib/python3.5/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/zorgan/postr/env/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/zorgan/postr/env/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/home/zorgan/postr/env/lib/python3.5/site-packages/dbbackup/utils.py", line 116, in wrapper
    func(*args, **kwargs)
  File "/home/zorgan/postr/env/lib/python3.5/site-packages/dbbackup/management/commands/dbbackup.py", line 61, in handle
    self._save_new_backup(database)
  File "/home/zorgan/postr/env/lib/python3.5/site-packages/dbbackup/management/commands/dbbackup.py", line 88, in _save_new_backup
    self.write_to_storage(outputfile, filename)
  File "/home/zorgan/postr/env/lib/python3.5/site-packages/dbbackup/management/commands/_base.py", line 88, in write_to_storage
    self.storage.write_file(file, path)
  File "/home/zorgan/postr/env/lib/python3.5/site-packages/dbbackup/storage.py", line 82, in write_file
    self.storage.save(name=filename, content=filehandle)
  File "/home/zorgan/postr/env/lib/python3.5/site-packages/django/core/files/storage.py", line 54, in save
    return self._save(name, content)
  File "/home/zorgan/postr/env/lib/python3.5/site-packages/storages/backends/s3boto3.py", line 452, in _save
    self._save_content(obj, content, parameters=parameters)
  File "/home/zorgan/postr/env/lib/python3.5/site-packages/storages/backends/s3boto3.py", line 467, in _save_content
    obj.upload_fileobj(content, ExtraArgs=put_parameters)
  File "/home/zorgan/postr/env/lib/python3.5/site-packages/boto3/s3/inject.py", line 513, in object_upload_fileobj
    ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
  File "/home/zorgan/postr/env/lib/python3.5/site-packages/boto3/s3/inject.py", line 431, in upload_fileobj
    return future.result()
  File "/home/zorgan/postr/env/lib/python3.5/site-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/home/zorgan/postr/env/lib/python3.5/site-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
  File "/home/zorgan/postr/env/lib/python3.5/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/home/zorgan/postr/env/lib/python3.5/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/home/zorgan/postr/env/lib/python3.5/site-packages/s3transfer/upload.py", line 692, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/home/zorgan/postr/env/lib/python3.5/site-packages/botocore/client.py", line 324, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/zorgan/postr/env/lib/python3.5/site-packages/botocore/client.py", line 622, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

Any idea what the problem is?

Answer 1

I solved this by adding permissions for s3:PutObjectAcl to the IAM policy.

Recent versions of boto3 & django-storages (which django-dbbackup uses) set the default ACL per object during each PutObject operation. So you need permissions for putting the object and updating the ACL.

Here's an example policy based on the one in the question:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Allow All",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3:::****-bucket/*"
        }
    ]
}