CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-storage
Brayden
Brayden

Reputation: 155

Is it possible to have a race condition when composing into the same destination object in GCS

Let's say I have the following scenario:

There are two processes running at the exact same time. Both create temporary files and attempt to append them, via compose, to a shared destination file.

Process 1: creates a temp file (my_file1.txt) and does compose(shared_file.txt, my_file1.txt) -> shared_file.txt
Process 2: creates a temp file (my_file2.txt) and does compose(shared_file.txt, my_file2.txt) -> shared_file.txt

My hope is that shared_file.txt contains the content of my_file1.txt and my_file2.txt.

Is it possible that the result of shared_file.txt will contain only one of the temp files? Are there consistency checks within Google Cloud Storage that ensure that compose won't overwrite a file if it's changed during the composition?

If Google Cloud Storage locks the destination file during composition or linearly executes requests, I'd expect it always contains both my_file1.txt and my_file2.txt but with no guaranteed order. Otherwise, I'd expect that I'd need to use Generation Matching+Request retries to ensure both temp files are appended.

Upvotes: 0

Views: 396

Answers (1)

John Hanley
John Hanley

Reputation: 81424

The Compose API has an optional query parameter to control object creation:

Setting ifGenerationMatch to 0 makes the operation succeed only if there is no live destination object

Objects: compose

In your example, if process 1 succeeds, process 2 will fail if you specify the ifGenerationMatch header correctly. The reverse can be true pending which process completes first.

Upvotes: 0

Related Questions