CODEWITHSUNDEEP
google-app-engine
Dan
Dan

Reputation: 5313

How to fan out URL Fetch requests in a timely fashion?

Every minute or so my app creates some data and needs to send it out to more than 1000 remote servers via URL Fetch callbacks. The callback URL for each server is stored on separate entities. The time lag between creating the data and sending it to the remote servers should be roughly less than 5 seconds.

My initial thought is to use the Pipeline API to fan out URL Fetch requests to different task queues.

Unfortunately task queues are not guaranteed to be executed in a timely fashion. Therefore from requesting a task queue start to it actually executing could take minutes to hours. From previous experience this gap is regularly over a minute so is not necessarily appropriate.

Is there any way from within App Engine to achieve what I want? Maybe you know of an outside service that can do the fan out in a timely fashion?

Upvotes: 1

Views: 364

Answers (2)

jkflying
jkflying

Reputation: 1089

How about using the async API? You could then do a large number of simultaneous URL calls, all from a single location.

If the performance is particularly sensitive, you could do them from a backend and use a B8 instance.

Upvotes: 1

T. Steinrücken
T. Steinrücken

Reputation: 469

Well, there's probably no good solution for the gae here. You could keep a backend running; hammering the datastore/memcache every second for new data to send out, and then spawn dozens of async url-fetches. But thats really inefficient...

If you want a 3rd party service, pubnub.com is capable of doing fan-out, however i don't know if it could fit in your setup.

Upvotes: 1

Related Questions