CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-storage
barnabus
barnabus

Reputation: 882

Google Cloud Storage Transfer

We’ve been using Google Cloud Storage Transfer service and in our data source (AWS) we had a directory accidentally deleted, so we figured it would be in the data sink however upon taking a looking it wasn’t there despite versioning being on.

This leads us to believe in Storage Transfer the option deleteObjectsUniqueInSink hard deletes objects in the sink and removes them from the archive.

We'e been unable to confirm this in the documentation.

Is GCS Transfer Service's deleteObjectsUniqueInSink parameter in the TransferSpec mutually exclusive with GCS's object versioning soft-delete?

Upvotes: 0

Views: 559

Answers (1)

Joseph Cox
Joseph Cox

Reputation: 81

When the deleteObjectsUniqueInSink option is enabled, Google Cloud Storage Transfer will

  1. List only the live versions of objects in source and destination buckets.
  2. Copy any objects unique in the source to the destination bucket.
  3. Issue a versioned delete for any unique objects in the destination bucket.

If the unique object is still live at the time that Google Cloud Storage Transfer issues the deletion, it will be archived. If another process, such as Object Lifecycle Management, archived the object before the deletion occurs, the object could be permanently deleted at this point rather than archived.

Edit: Specifying the version in the delete results in a hard delete (Objects Delete Documentation), so transfer service is currently performing hard deletes for unique objects. We will update the service to instead perform soft deletions.

Edit: The behavior has been changed. From now on deletions in versioned buckets will be soft deletes rather than hard deletes.

Upvotes: 2

Related Questions