CODEWITHSUNDEEP
javagoogle-app-engineurlupload
Tim Niblett
Tim Niblett

Reputation: 1187

Can I upload an external URL containing a lot of data to App Engine blob store?

I'd like to upload external images (for which I have the URL) into the Blobstore. I can't post the data, as its on another web site.

Its straightforward to upload by (a) using the fetch service to fetch the URL and then (b) using the file service to store the data from the URL as a blob. However this requires loading the data into my running instance. As I understand the fetch service all the data has to reside in memory before it can be uploaded, which restricts the maximum data size. If the fetch service streamed data that wouldn't be too bad.

Does anyone know a way I can copy an external image into a blob in a memory-efficient way?

Tim

Upvotes: 2

Views: 123

Answers (1)

user479870
user479870

Reputation:

If the server supports it, you can download the file in chunks. Usually, servers indicate this by sending the Accept-Ranges header. You can request a specific byte range by setting the Range header. If the server accepts this and only serves the specified byte range, it sends the Content-Range header with the request.

In theory, you should then be able to keep appending to the file object, until you finalize it.

Upvotes: 2

Related Questions