CODEWITHSUNDEEP
rubysidekiq
Maxime
Maxime

Reputation: 138

How can I create sidekiq queues with variable names at runtime?

I am trying to create queues with variable queue names.

queue_name = "guide_" + guide['id'].to_s

Sidekiq::Client.push({
    'class' => GuidePdfWorker,
    'queue' => queue_name,
    'args'  => [key],
    'backtrace' => true
})

I know that I am supposed to add them to config/sidekiq.yml, but I can't, since I don't know the value of queue_name.

When I log Sidekiq::Client.registered_queues() I can see my queues, but they are never processed.

Upvotes: 6

Views: 7907

Answers (3)

haffla
haffla

Reputation: 1066

The gem suggested by the accepted answer hasn't received any updates in the last nine years. I found the gem sidekiq_limit-fetch backed by Evil Martians and just verified that it supports dynamic queues that do not have to be specified when Sidekiq starts.

Check out the dynamic queues option. Now you can do

SomeWorker.set(queue: "your-dynamic-queue").perform_async(args)

Sidekiq will fetch this job from the "your-dynamic-queue" queue and execute it.

Upvotes: 0

Kache
Kache

Reputation: 16727

Just to provide a more complete, updated answer: there are plugins and extensions to Sidekiq that can do things like this, but Sidekiq is not designed to operate this way.

I don't recommend having more than a handful of queues ... and Sidekiq Pro cannot reliably handle multiple queues without polling

https://github.com/mperham/sidekiq/wiki/Advanced-Options#queues

the number of named queues used should be minimized.

https://github.com/mperham/sidekiq/issues/835

Instead, consider having a known, static, queue with a worker that dispatches based on what you'd like to be dynamic:

class GuidePdfWorker
  include Sidekiq::Worker

  sidekiq_options queue: 'default'

  def perform(guide)
    # branch on guide['id'], perhaps re-queue in higher or lower priority queue?
  end
end

Upvotes: 0

davogones
davogones

Reputation: 7399

The Sidekiq Dynamic Queues gem will probably help you.

Upvotes: 3

Related Questions