CODEWITHSUNDEEP
google-app-engineurlfetch
speedplane
speedplane

Reputation: 16141

App Engine: Retrieving Large Files from Third Party Site

In my Google App Engine application, a task has to download a large file from a third party site, process it, and store the results in the datastore/blobstore. Given that urlfetch can only receive 32MB of data in a response, and this file is larger than 32MB, what is the best way of doing this?

The only solution I can think of is to build another server not subject to the urlfetch limits and then have the server download the large file and upload it to a GAE blobstore upload url. This seems like a complete headache to implement, and having a second server defeats the purpose of using GAE. Isn't there an easier way of getting large files into Google App Engine?

Upvotes: 0

Views: 831

Answers (1)

Faisal
Faisal

Reputation: 2276

You can try the new sockets api: https://developers.google.com/appengine/docs/python/sockets/overview#limitations-and-restrictions

It says if you want to get around the request limit you need to upload your own httplib. I can't think of reason why this won't work yet, run it in a backends to avoid timeout and writing chunks of data as you recieve it to google cloud storage or blobstore (but its files api is depracated).

But other than that yeah there is no other way than what Deviling Master said and your another server that handles it.

Upvotes: 0

Related Questions