CODEWITHSUNDEEP
rabbitmqmessage-queuezeromqbeanstalkd
catalinux
catalinux

Reputation: 1452

Limiting a queue over time

I'm using an API that is limited for usage, let's say: no more than 10 calls per second, and no more than 5000 calls per day.

I am handling this calls in a beanstalkd queue process job. How can I limit the processing of this jobs, having in mind the API's limits.

Upvotes: 4

Views: 670

Answers (2)

Andras
Andras

Reputation: 3055

If all workers share durable state, they can update shared status and collective implement rate limiting.

If the only shared writable state is the queue itself, you could create ticketing tubes for the rate limited jobs, and have a rate limit manager insert tickets (permission slips) to control when the jobs get run. Would need changes to the workers, would need a way to time out unused tickets, but should be workable.

Edit: a "valid until" timestamp in the ticket might do it for per-second limits. Per-day limits might need a feedback tube back to let the rate limit manager know about actual usage (to implement a rolling 24 hour window instead of the 5000 all getting reset at midnight)

Upvotes: 0

Pentium10
Pentium10

Reputation: 208002

When you use Beanstalkd you can have the tube paused for a certain seconds.

When you reserve a job, and you know the API call failed during that call, you get to pause the tube for X seconds.

You can find out the time needed to pause the tube, either from your API response (usually they return you are locked until Time X), or start with something adaptive like pause for the next 60 seconds, and increase/decrease on the go.

If you know you can delay, or disperse in advance, before placing the jobs into your queues, you can also add a delay to the job, so it won't execute immediately, this way you can have your jobs distributed over time.

Also there is a great post about distributed rate limiting using redis

Upvotes: 2

Related Questions