Dependent jobs in Sidekiq - How to avoid the one that launches to finish?

Question

I have the following job structure in my Rails app:

def perform(page_ids, email, order_id)
    job_ids = []
    page_ids.each do |id|
      job_ids << ShoeBuilder.perform_async(id, order_id)
    end
    CheckStatus.perform_async(job_ids, email) unless job_ids.empty?
end

Basically I launch N ShoeBuilders, and I have a CheckStatus worker that keeps checking if all of those jobs have finished in order to send an email to the User. Here is how CheckStatus looks:

 def perform(ids, email, order_id)
    finished = []
    while finished.size < ids.size 
      ids.each do |id|
        status = Sidekiq::Status::status(id)
        # We consider finished if it has been completed or failed
        if (( Sidekiq::Status::complete?(id) || Sidekiq::Status::failed?(id) ) && !finished.include?(id))
          finished << id
        end
      end
      sleep 5 
    end
    Notifier.thanks(email, order_id).deliver
  end

So, my problem here is the following:

The ShoeBuilder does some work, but at some point, if it has not been able to find its information (it is a parser worker), will launch another instance of ShoeBuilder. And here comes my problem:

• When I launch a ShoeBuilder job from within another ShoeBuilder job, the first one finishes. How can I avoid that?
• If I can't avoid that, how can I tell the CheckStatus, to take into account this new job id, instead of the original one?

Answer 1

If you're not able to use Sidekiq Pro, Sidekiq Superworker also supports this:

# config/initializers/superworkers.rb
Superworker.create(:MySuperworker, :page_ids, :email, :order_id) do
  batch page_ids: :page_id do
    ShoeBuilder :page_id, :order_id
  end
  Notifier :email, :order_id
end

# Anywhere
MySuperworker.perform_async([1, 2, 3], 'foo@gmail.com', 378)

When you run MySuperworker, it will queue ShoeBuilder jobs for all of the page IDs. When all of those have finished, Notifier will run. (Disclaimer: I'm the gem's author.)

Answer 2

Sidekiq Pro's Batch feature handles this easily.