CODEWITHSUNDEEP
rubybeanstalkd
persnickety
persnickety

Reputation: 13

Reserving multiple jobs from a beanstalkd queue

Is there a way I can reserve multiple jobs from a beanstalkd queue at once?

I'm making requests to an external API that can return up to 10 results per query. They limit the number of requests I can make each day, so the more results I get per request the better.

I couldn't find any mention of this functionality in the documentation so I'm using this workaround. Does anyone know of a better way to achieve this? Or a more appropriate tool for the job than beanstalkd perhaps?

loop do
  sleep(0.3)

  while @beanstalk.tubes[example].peek(:ready)
    jobs = []

    catch(:done) do
      10.times do |i|
        if @beanstalk.tubes[example].peek(:ready) then
          job = @beanstalk.tubes[example].reserve(0)
          jobs << job.body
          job.delete
        else
          throw(:done)
        end
      end
    end

    process(jobs)

  end
end

Upvotes: 1

Views: 1394

Answers (1)

Keith Rarick
Keith Rarick

Reputation: 281

You can reserve several jobs concurrently by calling reserve several times in a row before deleting or releasing those jobs.

Based on the code sample you provided, it could look something roughly like this:

loop do
  timeout = nil
  jobs = []

  begin
    10.times do |i|
      jobs << @beanstalk.tubes[example].reserve(timeout)
      timeout = 0
    end
  rescue Beaneater::TimedOutError
    # nothing to do
  end

  process(jobs.map{|j| j.body})

  jobs.map do |job|
    job.delete
  end
end

Upvotes: 4

Related Questions