Understanding object deletion with versioning in S3 with lifecycle

Question

Following policy states to delete the current version of an object after 30 days and previous version after 30 days.

Now suppose I upload an object on 1st April in a version enabled bucket and then upload same object on 10th April.

If I had not uploaded second version, current object would have got deleted on 30th April.

So my question is what would happen in case I have uploaded second version on 10th april.

Would new version and old version be deleted both on 10th May OR, old version gets deleted on 30th April and new version get's deleted on 10th May?

{
        "Rules": [{
                        "ID": "DeletionOfFileBasedOnQATag",

                        "Status": "Enabled",
                        "Expiration": {
                                "Days": 30
                        },
                        "NoncurrentVersionExpiration": {
                                "NoncurrentDays": 30
                        }
                }
        ]
}

Answer 1

NewerNoncurrentVersions is a new S3 version feature. This can help you retain a number of versions. This works with aws-cli v2 or Terraform provider 4.41.0.

NoncurrentDays: Number of days an object is noncurrent before Amazon S3 can perform the associated action. If you want to avoid any expiration of the current version of the files in the current/ folder, ensure that there is no Expiration attribute included in the rule.

"Expiration": {
        "Days": ??
}

Create a JSON file called lifecycle.json that defines the desired lifecycle rule. In below case, the rule will be applied only to versioned data in the current/ folder.

cat <<EOF > ./lifecycle.json
{
  "Rules": [
    {
      "Status": "Enabled",
      "Filter": {
        "Prefix": "current/"
      },
      "NoncurrentVersionExpiration": {
        "NoncurrentDays": 1,
        "NewerNoncurrentVersions": 60
      },
      "ID": "Keep_60_copies_of_versioned_data"
    }
  ]
}
EOF

/usr/local/bin/aws s3api put-bucket-lifecycle-configuration --bucket my-bucket --lifecycle-configuration file://./lifecycle.json --output json

Answer 2

Before to trying to answer your question let me clear some basics I understand:

Imagine your S3 file which is versioned as a stack where inside contains a set of files with currentversion and 0 or multiple noncurrenversion. Whenever a new update happens to the currentversion, the new version is stack on the top and becomes currentversion while the rest becomes a set of noncurrentversion in order (since they behave like a stack).

Despite your S3 configuration doesn't have versioning enabled, the same schema is being used. In those cases the noncurrenversion is set to 0 and the versioning mechanism is disabled.

Consider that each object of this "stack" in the S3 file contains a time mark that it is set at the time when the object is being created and added to the top of the stack.

Based on the first section of your policy:

                    "Expiration": {
                            "Days": 30
                    }

Each time this rule is being triggered it will read the time mark of the currentversion and if it is older than 30 days it will be removed. It has no impact on noncurrent object versions. Moreover the last

Source: AWS - Elements to describe lifecycle actions in versioned buckets - Expiration action element - Versioned bucket

Based on the second section of your policy:

                    "NoncurrentVersionExpiration": {
                            "NoncurrentDays": 30
                    }

By reading into the AWS Documentation of Lifecycle rules based on object's age. When specifying the number of days in the NoncurrentVersionTransition and NoncurrentVersionExpiration actions in a Lifecycle configuration, note the following:

It is the number of days from when the version of the object becomes noncurrent (that is, when the object is overwritten or deleted), that Amazon S3 will perform the action on the specified object or objects.

Whenever a new version is being put on the top of the stack. The timemark will be updated at the time when this action happened.

Each time this rule is being triggered it will read the time mark of all the noncurrent version objects in the stack and if any object is older than 30 days it will be removed.

Conclusion

Would new version and old version be deleted both on 10th May OR, old version gets deleted on 30th April and new version get's deleted on 10th May?

Both versions will be deleted at 10 of May (10 April + 30 days). Because the new version will have a time mark that will start to count from 10 of May but also the old version will be deleted at the same time (if nothing else happens in the meantime), because when the new version is created the time mark the old one is being updated as well. Both at the same moment indeed.

I hope this helps. And thanks for the corrections in the bellow comments.

Answer 3

Based on the excerpts below from AWS Docs, in the current case, both objects (latest version and previous version) will get deleted on 10th May

Current version will expire on 10th May, since it's created on 10th April (we have expiration = 30days)

Noncurrent version will expire on 10th May, since it's also created=modified on 10th April (and we have non-current-expiration = 30days)

Amazon S3 maintains only the last modified date for each object. For example, the Amazon S3 console shows the Last Modified date in the object Properties pane. When you initially create a new object, this date reflects the date the object is created. If you replace the object, the date changes accordingly. So when we use the term creation date, it is synonymous with the term last modified date. ref:https://docs.aws.amazon.com/AmazonS3/latest/dev/intro-lifecycle-rules.html#intro-lifecycle-rules-number-of-days

NoncurrentVersionExpiration action element – Use this action to specify how long (from the time the objects became noncurrent) you want to retain noncurrent object versions before Amazon S3 permanently removes them. The deleted object can't be recovered.

This delayed removal of noncurrent objects can be helpful when you need to correct any accidental deletes or overwrites. For example, you can configure an expiration rule to delete noncurrent versions five days after they become noncurrent. For example, suppose that on 1/1/2014 10:30 AM UTC, you create an object called photo.gif (version ID 111111). On 1/2/2014 11:30 AM UTC, you accidentally delete photo.gif (version ID 111111), which creates a delete marker with a new version ID (such as version ID 4857693). You now have five days to recover the original version of photo.gif (version ID 111111) before the deletion is permanent. On 1/8/2014 00:00 UTC, the Lifecycle rule for expiration executes and permanently deletes photo.gif (version ID 111111), five days after it became a noncurrent version. https://docs.aws.amazon.com/AmazonS3/latest/dev/intro-lifecycle-rules.html#intro-lifecycle-rules-actions