Status on AWS S3 cross region replication delete operations behaviour

Question

I've been surprised to find out that file deletion was not replicated in a S3 bucket Cross Region Replication situation, running this simple test:

1. simplest configuration of a CRR
2. upload a new file
3. check it is replicated
4. delete the file (not a version of the file)

So I checked the documentation and I find this statement:

If you delete an object from the source bucket, the following occurs:

• If you make a DELETE request without specifying an object version ID, Amazon S3 adds a delete marker. Amazon S3 deals with the delete marker as follows:
  • If using latest version of the replication configuration, that is you specify the Filter element in a replication configuration rule, Amazon S3 does not replicate the delete marker.
  • If don't specify the Filter element, Amazon S3 assumes replication configuration is a prior version V1. In the earlier version, Amazon S3 handled replication of delete markers differently. For more information, see Backward Compatibility .

The later link to backward compat tell me that:

• When you delete an object from your source bucket without specifying an object version ID, Amazon S3 adds a delete marker. If you use V1 of the replication configuration XML, Amazon S3 replicates delete markers that resulted from user actions.[...] In V2, Amazon S3 doesn't replicate delete markers and therefore you must set the DeleteMarkerReplication element to Disabled.

So if I sum this up:

• CRR configuration is considered v1 if there is no Filter
• with CRR configuration v1, file deletion is replicated, not with v2

Well, this is my configuration :

{
    "ReplicationConfiguration": {
        "Role": "arn:aws:iam::271226720751:role/service-role/s3crr_role_for_mybucket_to_myreplica",
        "Rules": [
            {
                "ID": "first replication rule",
                "Status": "Enabled",
                "Destination": {
                    "Bucket": "arn:aws:s3:::myreplica"
                }
            }
        ]
    }
}

And deletion is not replicated. So it makes me think that my configuration is still considered V2 (even if I have no filter).

---

So, can someone confirm this presumption? And could someone tell me what does:

In V2, Amazon S3 doesn't replicate delete markers and therefore you must set the DeleteMarkerReplication element to Disabled

really mean?

Answer 1

There are two different configuration when replicating delete marker, V1 and V2.

Currently, when you enable S3 Replication (CRR or SRR) from the console, V2 configuration is enabled by default. However, if your use case requires you to delete replicated objects whenever they are deleted from the source bucket, you need the V1 configuration

Here is the difference between V1 and V2:

• V1 configuration

The delete marker is replicated (V1 configuration). A subsequent GET request to the deleted object in both the source and the destination bucket does not return the object.

• V2 configuration

The delete marker is not replicated (V2 configuration). A subsequent GET request to the deleted object returns the object only in the destination bucket.

To enable V1 configuration (to replicate delete marker), use the policy below. Keep in mind that certain replication features such as tag-based filtering and Replication Time Control (RTC) that are only available in V2 configurations.

{
    "Role": " IAM-role-ARN ",
    "Rules": [
        {
            "ID": "Replication V1 Rule",
            "Prefix": "",
            "Status": "Enabled",
            "Destination": {
                "Bucket": "arn:aws:s3:::<destination-bucket>"
            }
        }
    ]
}

Here is the blog that describes these behavior in details: https://aws.amazon.com/blogs/storage/managing-delete-marker-replication-in-amazon-s3/

Answer 2

The issue comes from still not clear documentation from AWS. To use DeleteMarkerReplication, you need V1 of the configuration. To let AWS know that you want V1, you need to specify a Prefix element in your configuration, and no DeleteMarkerReplication element, so your first try was almost correct.

{
"ReplicationConfiguration": {
    "Role": "arn:aws:iam::271226720751:role/service-role/s3crr_role_for_mybucket_to_myreplica",
    "Rules": [
        {
            "ID": "first replication rule",
            "Prefix": "",
            "Status": "Enabled",
            "Destination": {
                "Bucket": "arn:aws:s3:::myreplica"
            }
        }
    ]
}

}

And of course you need the s3:ReplicateDelete permission in your policy.

Answer 3

Adding a comment as an answer because I cannot comment to @john-eikenberry's answer. I have tested answer suggested by John (Action "s3:ReplicateDelete") but it is not working.

Edit: A failed attempt:

I have also tried to put bucket replication with delete marker enabled but it failed. Error message is:

An error occurred (MalformedXML) when calling the PutBucketReplication operation: The XML you provided was not well-formed or did not validate against our published schema

Experiment details:

Existing replication configuration:

aws s3api get-bucket-replication --bucket my-source-bucket > my-source-bucket.json

{
    "Role": "arn:aws:iam::account-number:role/s3-cross-region-replication-role",
    "Rules": [
        {
            "ID": " s3-cross-region-replication-role",
            "Priority": 1,
            "Filter": {},
            "Status": "Enabled",
            "Destination": {
                "Bucket": "arn:aws:s3:::my-destination-bucket"
            },
            "DeleteMarkerReplication": {
                "Status": "Disabled"
            }
        }
    ]
}

aws s3api put-bucket-replication --bucket my-source-bucket --replication-configuration file://my-source-bucket-updated.json

{
    "Role": "arn:aws:iam::account-number:role/s3-cross-region-replication-role",
    "Rules": [
        {
            "ID": " s3-cross-region-replication-role",
            "Priority": 1,
            "Filter": {},
            "Status": "Enabled",
            "Destination": {
                "Bucket": "arn:aws:s3:::my-destination-bucket"
            },
            "DeleteMarkerReplication": {
                "Status": "Enabled"
            }
        }
    ]
}

Answer 4

I believe I've figured this out. It looks like whether the Delete Markers are replicated or not depends on the permissions in the Replication Role.

If your replication role has the permission s3:ReplicateDelete on the destination, then Delete Markers will be replicated. If if does not have that permission they are not.

Below is the Cloudformation YAML for my Replication role with the ReplicateDelete permission commented out as an example. With this setup it does not replicate Delete Markers, uncomment the permission and it will. Note the permissions is based on what AWS actually creates if you set up the replication via the console (and they differ slightly from those in the documentation).

ReplicaRole:
  Type: AWS::IAM::Role
  Properties:
    #Path: "/service-role/"
    AssumeRolePolicyDocument:
      Version: '2012-10-17'
      Statement:
      - Effect: Allow
        Principal:
          Service:
          - s3.amazonaws.com
        Action:
        - sts:AssumeRole
    Policies:
    - PolicyName: "replication-policy"
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Resource:
              - !Sub "arn:aws:s3:::${LiveBucketName}"
              - !Sub "arn:aws:s3:::${LiveBucketName}/*"
            Action:
              - s3:Get*
              - s3:ListBucket
          - Effect: Allow
            Resource: !Sub "arn:aws:s3:::${LiveBucketName}-replica/*"
            Action:
              - s3:ReplicateObject
              - s3:ReplicateTags
              - s3:GetObjectVersionTagging
              #- s3:ReplicateDelete

Answer 5

I have seen exactly the same behaviour. I was unable to create a v1 situation to get DeleteMarker replication to occur.