CODEWITHSUNDEEP
google-app-enginegoogle-cloud-datastore
Kevin
Kevin

Reputation: 67

Google App Engine - how to transfer large files from datastore to blobstore?

I've got a bunch of large files in the regular cloud storage that I'd like to programmatically move over to the blobstore for further processing using the mapreduce library. (Since there is a BlobstoreLineInputReader but not a Datastore version.) I've tried making a url for the gs file and having the blobstore try reading it in itself, also I've tried buffered reads, but for large files I still hit a memory error. (I avoid the deadline exceed error (more 60 seconds) for blobstore files by opening in append mode and finalizing only at the end.) It seems like there should be an efficient way to do this since both the datastore and blobstore are part of the same application context, but I haven't found it.

Upvotes: 0

Views: 435

Answers (1)

Stuart Langley
Stuart Langley

Reputation: 7054

I'm confused because you mention cloud storage and datastore almost interchangeably here.

If your data is in Google Cloud storage then you can create BlobKeys for the files and use them with any current Blobstore API.

i.e. 

blobkey = blobstore.create_gs_key('/gs/my_bucket/my_object').

If your files are in the datastore then you'll need to use the files API to move them to Cloud Storage/Blobstore and then process them from there.

Upvotes: 4

Related Questions