CODEWITHSUNDEEP
google-app-enginegoogle-cloud-storage
Tom Fishman
Tom Fishman

Reputation: 1826

How to append write to google cloud storage file from app engine?

This deprecated API supports appending content to google cloud storage file:

FileWriteChannel FileService.openWriteChannel(AppEngineFile file,
                                boolean lock)

But, 

GcsOutputChannel GcsService.createOrReplace(GcsFilename filename,
                               GcsFileOptions options)

doesn't seem so.

What is the solution here?

Upvotes: 19

Views: 24086

Answers (3)

Paul Fabbroni
Paul Fabbroni

Reputation: 123

Very old question, but you can probably do this now by leveraging the combine function (save your file into a bunch of small ones as you read/write your data and then combine them all at the end) - https://googleapis.dev/nodejs/storage/latest/Bucket.html#combine

Upvotes: 0

Ezequiel Muns
Ezequiel Muns

Reputation: 7762

While Google Cloud Storage objects are immutable, there is something that could help you if you need to do limited appends: object compose

This operation basically concatenates the contents of a number of objects in the same bucket under a new name, like cat file1 file2 > newfile but without re-writing data.

So you could create a new object, upload the contents to append to it, close and subsequently compose this new piece at the end of your main file.

There is a caveat: a maximum of 1024 'pieces' per object are allowed, and composing objects that are themselves composed counts as composing the original pieces (so you can't cheat the limit by iterated composing).

Upvotes: 17

norlesh
norlesh

Reputation: 1861

Sorry to tell you but it cannot be done without replacing the file each time.

As stated in the Cloud Services docs Under Object immutability

Objects are immutable, which means that an uploaded object cannot change throughout its storage lifetime. An object's storage lifetime is the time between successful object creation (upload) and successful object deletion. In practice, this means that you cannot make incremental changes to objects, such as append operations or truncate operations. However, it is possible to overwrite objects that are stored in Google Cloud Storage because an overwrite operation is in effect a delete object operation followed immediately by an upload object operation. So a single overwrite operation simply marks the end of one immutable object's lifetime and the beginning of a new immutable object's lifetime.

Upvotes: 19

Related Questions