CODEWITHSUNDEEP
rubyredisdelayed-jobresque
meso_2600
meso_2600

Reputation: 2130

How resque checks when to run a job?

I have found the Resque:

https://github.com/elucid/resque-delayed

And I can see that I can schedule delayed Job. My question is, how does it check for delayed jobs? If I have 5000 delayed jobs in one month time, I hope it doesn't check every 10 seconds all delayed jobs.

So how is it being done?

Upvotes: 3

Views: 167

Answers (1)

Didier Spezia
Didier Spezia

Reputation: 73206

It does not have to check all the delayed jobs. It maintains a sorted set in Redis, the jobs being sorted by their scheduled time. See the code at:

https://github.com/elucid/resque-delayed/blob/master/lib/resque-delayed/resque-delayed.rb

Each time the daemon awakes, only the first item of the set needs to be checked (using a ZRANGEBYSCORE command). The daemon fetches the relevant jobs one by one, until the polling query returns no result, then it sleeps again.

Performance could be further improved by fetching the jobs n by n. It could be implemented using a server-side Lua script as a polling query:

local res = redis.call('ZRANGEBYSCORE',KEYS[1], "-inf", ARGV[1], 'LIMIT', 0, 10 )
if #res > 0 then
   redis.call( 'ZREMRANGEBYRANK', KEYS[1], 0, #res-1 )
   return res
else
   return false
end

In one roundtrip, this script gets 10 jobs (if available), and delete them from the zset. Much better than the 11 ZRANGEBYSCORE and 10 ZREM, currently required by Resque-delayed.

Upvotes: 1

Related Questions