Restoring files on a version enabled amazon s3 bucket

Question

I am trying to enable versioning and lifecycle policies on my Amazon S3 buckets. I understand that it is possible to enable Versioning first and then apply LifeCycle policy on that bucket. If you see the image below, that will confirm this idea.

I have then uploaded a file several times which created several versions of the same file. I then deleted the file and still able to see several versions. However, if I try to restore a file, I see that the Initiate Restore option is greyed out.

I would like to ask anyone who had any similar issue or let me know what I am doing wrong.

Thanks,

Answer 1

If you have a lot of deleted files to restore. You might want to use a script to do the job for you.

The script should

1. Get the versions of objects in your bucket using the Get object versions API
2. Inspect the versions data to get Delete Marker (i.e. delete objects) name and version id
3. Delete the markers found using the marker names and version ids using Delete object API

Python example with boto:

This example script deletes delete markers found one by one once.

#!/usr/bin/env python
import boto

BUCKET_NAME = "examplebucket"
DELETE_DATE = "2015-06-08"

bucket = boto.connect_s3().get_bucket(BUCKET_NAME)

for v in bucket.list_versions():
    if (isinstance(v, boto.s3.deletemarker.DeleteMarker) and
            v.is_latest and
            DELETE_DATE in v.last_modified):
        bucket.delete_key(v.name, version_id=v.version_id)

Python example with boto3:

However, if you have thousands of objects, this could be a slow process. AWS does provide a way to batch delete objects with a maximum batch size of 1000.

The following example script searches your objects with a prefix, and test them if they are deleted ( i.e. current version is a delete marker ) and them batch delete them. It is set to search 500 objects in your bucket in each batch, and try to delete multiple object with a batch no more than 1000 objects.

import boto3

client = boto3.client('s3')


def get_object_versions(bucket, prefix, max_key, key_marker):
    kwargs = dict(
        Bucket=bucket,
        EncodingType='url',
        MaxKeys=max_key,
        Prefix=prefix
    )

    if key_marker:
        kwargs['KeyMarker'] = key_marker

    response = client.list_object_versions(**kwargs)

    return response


def get_delete_markers_info(bucket, prefix, key_marker):
    markers = []
    max_markers = 500
    version_batch_size = 500

    while True:
        response = get_object_versions(bucket, prefix, version_batch_size, key_marker)
        key_marker = response.get('NextKeyMarker')
        delete_markers = response.get('DeleteMarkers', [])

        markers = markers + [dict(Key=x.get('Key'), VersionId=x.get('VersionId')) for x in delete_markers if
                             x.get('IsLatest')]

        print '{0} -- {1} delete markers ...'.format(key_marker, len(markers))

        if len(markers) >= max_markers or key_marker is None:
            break

    return {"delete_markers": markers, "key_marker": key_marker}


def delete_delete_markers(bucket, prefix):
    key_marker = None

    while True:
        info = get_delete_markers_info(bucket, prefix, key_marker)
        key_marker = info.get('key_marker')
        delete_markers = info.get('delete_markers', [])

        if len(delete_markers) > 0:
            response = client.delete_objects(
                Bucket=bucket,
                Delete={
                    'Objects': delete_markers,
                    'Quiet': True
                }
            )

            print 'Deleting {0} delete markers ... '.format(len(delete_markers))
            print 'Done with status {0}'.format(response.get('ResponseMetadata', {}).get('HTTPStatusCode'))
        else:
            print 'No more delete markers found\n'
            break


delete_delete_markers(bucket='data-global', prefix='2017/02/18')

Answer 2

With the new console, you can do it as following.

1. Click on the Deleted Objects button
2. You will see your deleted object below, Select it
3. Click on More -> Undo delete

Answer 3

Bucket Versioning on Amazon S3 keeps all versions of objects, even when they are deleted or when a new object is uploaded under the same key (filename).

As per your screenshot, all previous versions of the object are still available. They can be downloaded/opened in the S3 Management Console by selecting the desired version and choosing Open from the Actions menu.

If Versions: Hide is selected, then each object only appears once. Its contents is equal to the latest uploaded version of the object.

Deleting an object in a versioned bucket merely creates a Delete Marker as the most recent version. This makes the object appear as though it has been deleted, but the prior versions are still visible if you click the Versions: Show button at the top of the console. Deleting the Delete Marker will make the object reappear and the contents will be the latest version uploaded (before the deletion).

If you want a specific version of the object to be the "current" version, either:

• Delete all versions since that version (making the desired version that latest version), or
• Copy the desired version back to the same object (using the same key, which is the filename). This will add a new version, but the contents will be equal to the version you copied. The copy can be performed in the S3 Management Console -- just choose Copy and then Paste from the Actions Menu.

Initiate Restore is used with Amazon Glacier, which is an archival storage system. This option is not relevant unless you have created a Lifecycle Policy to move objects to Glacier.

Answer 4

I have realised that I can perform and Initiate Restore operation once the object is stored on Gliacer, as shown by the Storage Class of the object. To restore a previous copy on S3, the Delete marker on the current Object has to be removed.