CODEWITHSUNDEEP
ruby-on-railsapi
Sharpie
Sharpie

Reputation: 17663

Throttling outbound API calls generated by a Rails app

I am not a professional web developer, but I like to wrench on websites as a hobby. Recently, I have been playing with developing a Rails app as a project to help me learn the framework. The goal of my toy app is to harvest data from another service through their API and make it available for me to query using a search function.

However, the service I want to pull data from imposes a rate limit on the number of API calls that may be executed per minute. I plan on having my app run a daily update which may generate a burst of API calls that far exceeds the limit provided by the external service. I wish to respect the performance of the external site and so would like to throttle the rate at which my app executes the calls.

I have done a little bit of searching and the overwhelming amount of tutorial material and pre-built libraries I have found cover throttling inbound API calls to a web app and I can find little discussion of controlling the flow of outbound calls.

Being both an amateur web developer and a rails newbie, it is entirely possible that I have been executing the wrong searches in the wrong places. Therefore my questions are:

I have some ideas of how I might go about writing a throttling system using a queue-based worker like DelayedJob or Resque to manage the API calls, but I would rather spend my weekends building the rest of the site if there is a good pre-built solution out there already.

Upvotes: 13

Views: 5106

Answers (3)

onli
onli

Reputation: 183

There is now a gem for that: throttle-queue. It takes a code block and makes sure it gets executed only x times per second. This is an example taken from the Readme, that would fetch only three files per second:

require 'throttle-queue'

q = ThrottleQueue.new 3
files.each {|file|
    q.background(file) {
        fetch file
    }
}

Upvotes: 3

Bartosz Pietrzak
Bartosz Pietrzak

Reputation: 146

Try using nginx as a proxy: http://codetunes.com/2011/07/26/outbound-api-rate-limits-the-nginx-way .

If you're on heroku, then consider using the Slow Web gem.

Upvotes: 9

user336851
user336851

Reputation: 161

The reason nobody talk about outbound throttling is that it's usually pretty trivial, since you control it. Controlling bandwidth can be a bit harder, but controlling number of request ?

ri Kernel#sleep

So, if you're allowed 10 api calls per min you just sleep(6) after each call

Upvotes: 0

Related Questions