CODEWITHSUNDEEP
ruby-on-railsrubysidekiq
damuz91
damuz91

Reputation: 1728

Sidekiq - Send job to Deadset directly without retrying

How to send a job to the Deadset?

I've been searching around and I see some other developers that want to Notify to their exception handling systems when a job fails, for example if the job connects to an unreliable third party API.

In my case I would like to send the job to the deadset right away instead of just discarding it or waiting around 20 days.I want A human kind to have to click the Retry or Delete button based on the failure reason in the Deadset UI.

So if I have this Russian Roulette job that would retry if feeling lucky or go to the graveyard (aka Deadset) if not, how would be a proper way to do it?

It has come to my mind to set the retry: 0 in the sidekiq_options but that wouldn't be the case always. I think one way would be to use the retry method and pass somehow the sidekiq_options retry: 0 so next time it retries it self then it goes to the Deadset.

class RussianRoulette < ActiveJobBase

  sidekiq_options retry: 10

  discard_on SomeException do |job, exception|
    puts "You are dead but somebody will check your corpse!"
    # How to send to deadset?
  end

  def perform(args)
    if feeling_lucky? # If this method fails it retries up to 10 times
      puts "You got another chance!"
      retry_job
    else
      puts "You are dead!"
      # What to do?:
      # throw :abort # No, because we want to send it to the deadset so somebody can erase or retry it
      # raise SomeException # Could be, but then how to send to deadset?
      # self.class.perform_later(args, {sidekiq_options: {retry: 0}}) # Is this even possible?
    end
  end
end

The closest I've achieved is to push directly in the Deadset using the Sidekiq API, from the source I see that it expects a 'message'; I've tried to push the job as a JSON but in the WebUI it seems the message parameter is not correct.

Sidekiq::DeadSet.new.kill(job.to_json) # Also tried job.arguments.to_json

enter image description here

Update:

I have followed the logic of the Kill button in the WebUI and I see there is a kill method that is called to Job instance (or not?) but I see it's not available when I call it manually on console.

(byebug) job.kill
*** NoMethodError Exception: undefined method `kill' for #<RussianRoulette:0x00007faf520805a8>

Upvotes: 2

Views: 1310

Answers (2)

damuz91
damuz91

Reputation: 1728

I ended up injecting directly in the Deadset the record by just using the params of my original job. I captured the exception (using a custom exception) and the rescue_from method.

class MyJob < ActiveJob::Base

  rescue_from(CustomException) do |exception|
    params = self.arguments
    params[0]['_aj_symbol_keys'] = params[0].keys.map(&:to_s)
    job = {
      "error_message": exception.message, 
      "error_class": exception.class.name, 
      "jid": SecureRandom.hex(12),
      "wrapped": self.class.name, 
      "created_at": Time.now.to_f, 
      "enqueued_at": Time.now.to_f, 
      "failed_at": Time.now.to_f, 
      "retry_count": 0,
      "args": [
        {
          "job_class": self.class.name,
          "job_id": SecureRandom.hex(12),
          "provider_job_id": nil,
          "priority": nil,
          "arguments":  params,
          "executions": 0,
          "queue_name": "default",
          "exception_executions": {},
          "locale": "en",
          "timezone": "UTC",
          "enqueued_at": Time.now.to_f
        }
      ],
      "retry": 0,
      "queue": "default",
      "class": "ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper"
    }
    Sidekiq::DeadSet.new.kill(job.to_json) 
  end

  def perform(params)
    raise CustomException.new("Here is a custom exception") if whatever
  end
end

So when i need to send a Job to the morgue directly inside my job logic i just raise an exception and thats it, the rescue_from block will capture it and create it in the morgue.

After this it shows up in my Morgue:

enter image description here

And some human can analyze it and delete or retry it. In my case this was a web scraper, so if the HTML of my target site goes invalid then a developer has to update the logic and we have to retry with the original arguments once the developer pushes the updated scraping code.

Upvotes: 1

yednamus
yednamus

Reputation: 583

One suggestion i can think of is split the queues in 2 different queues

  1. Queue with retry.
  2. Instant dead queue retry set to 0.

Now RussianRoulette class can enqueue to retry queue or dead queue based on the need. And also you can have a logic in dead queue to re enqueue back to the RussianRoulette or retry queue based on retry arg.

Upvotes: 1

Related Questions