CODEWITHSUNDEEP
amazon-web-servicesamazon-s3amazon-glacier
Dzhuneyt
Dzhuneyt

Reputation: 8691

Monitor bulk transitions from Glacier to Standard storage?

I have started a bulk S3 Glacier to Standard storage restoral operation using using the AWS CLI aws s3api restore-object command, with the intention of later downloading and copying files to a different backup location.

I've set the restore-request days to "1 day" to avoid being billed more than needed (AWS will delete the restored copy 1 day after it's been restored).

However, I am having difficulty understanding which of the thousands of files I requested a restore for - is now ready for downloading/copying. Is there an AWS API or an area in the interface where I can monitor current Glacier restorations - in bulk?

Upvotes: 0

Views: 835

Answers (1)

Michael - sqlbot
Michael - sqlbot

Reputation: 178956

aws s3api head-object allows you to poll an object to check its restoration status.

Find Restore in the output. This value comes from x-amz-restore, as documented in the HEAD Object API Action section of the S3 API Reference.

For an object that is ready to download, ongoing-request (whether the Glacier action is still pending) is set to false:

ongoing-request="false", expiry-date="Fri, 23 Dec 2012 00:00:00 GMT"

...otherwise, for objects not yet ready, it's set to true (the restore is ongoing), with no expiry date provided:

ongoing-request="true"

Note that variations of head-object are also available in the SDKs. In Node.JS it's s3.headObject()... in boto3, it's client.head_object()... etc.

Instead of polling, there's also a proactive "push" approach, using S3 Event Notifications.

Configuring an s3:ObjectRestore:Completed notifcation will generate messages every time a restored object is ready. You can optionally also use the s3:ObjectRestore:Post to capture the original request to restore the object (the requests which, in this case, you've already made).

If you enable these s3:ObjectRestore:Completed notifications, now, you should receive a notification event for each object, as it becomes ready, as long as the object was not already ready when you first set up the notification. You'll need to use the polling method, above, for that.

S3 event notifications generate messages that can be published to SNS Topics, sent to SQS Queues, or used as payload to invoke Lambda functions. If you are familiar with any of these technologies, it may be apparent how one of these might be useful for feeding a downstream process to perform the necessary follow-up actions.

Upvotes: 0

Related Questions