CODEWITHSUNDEEP
amazon-web-servicesamazon-s3storageamazon-glacier
Riccardo Casatta
Riccardo Casatta

Reputation: 1200

AWS storage: S3 object go to Glacier if never accessed in the last month

i am storing objects to S3, i would like that object never accessed in the last month go to glacier.

After some research i don't think i can achieve this, but i hope to be wrong.

When creating lifecycle for an s3 bucket the rule is based on object creation date (not last access date)

Setting the storage class for the object will not help according to http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html

"You cannot associate an object with the Glacier storage class as you upload it. You transition existing Amazon S3 objects to the Glacier storage class by using lifecycle management. For more information, see Object Lifecycle Management."

Does anyone know how can achieve this?

Thanks

Upvotes: 4

Views: 2761

Answers (4)

Aniket Kulkarni
Aniket Kulkarni

Reputation: 12983

It is not provided out of the box feature from S3 lifecycle rules. However, there is good AWS blog which explains the possible architecture to achieve the same.

Here is the URL -> expiring-amazon-s3-objects-based-on-last-accessed-date-to-decrease-costs
There are several other services are used apart from S3. These are Event bridge, Athena, Lambda. Here is the description from AWS blog:

  1. The S3 server access logs capture S3 object requests. These are generated and stored in the target S3 bucket.
  1. An S3 inventory report is generated for the source bucket daily. It is written to the S3 inventory target bucket.
  1. An Amazon EventBridge rule is configured that will initiate an AWS Lambda function once a day, or as desired.
  1. The Lambda function initiates an S3 Batch Operation job to tag objects in the source bucket. These must be expired using the following logic:
  • Capture the number of days (x) configuration from the S3 Lifecycle configuration.
  • Run an Amazon Athena query that will get the list of objects from the S3 inventory report and server access logs. Create a delta list with objects that were created earlier than 'x' days, but not accessed during that time.
  • Write a manifest file with the list of these objects to an S3 bucket.
  • Create an S3 Batch operation job that will tag all objects in the manifest file with a tag of "delete=True".
  1. The Lifecycle rule on the source S3 bucket will expire all objects that were created prior to 'x' days. They will have the tag given via the S3 batch operation of "delete=True".

Upvotes: 1

Spade Johnsson
Spade Johnsson

Reputation: 572

AFAIK, you can 'only' toy around with the lifecycle rules to set a fixed archiving date after object creation...

Upvotes: 0

Noman Ur Rehman
Noman Ur Rehman

Reputation: 6957

You cannot add conditional policies that apply to objects in S3 object lifecycle configuration which in your case is, is based on the object's last access time.

You can however transition objects to Glacier based on their age or on a specific date.

I would like to think you can handle it in your application but the s3 object returned does not have the last access time, if you use the the AWS SDK.

Details here

Upvotes: 4

Ronak Jain
Ronak Jain

Reputation: 2440

For associate Glacier storage class to objects you need to apply lifecycle transition rule on objects.

For example, If you applied a lifecycle rule to move everything under "folder1" into Glacier, Then this rule is applied to existing data as well as on newly uploaded data on "folder1". When data is moved to Glacier then you can see that the storage class becomes Glacier

When you delete a lifecycle rule then no new data will move into Glacier but old data will still sit in Glacier until you restore them.

For more detail : Amazon S3 - Object Lifecycle Management

(Disclosure : Bucket Explorer)

Upvotes: -1

Related Questions