CODEWITHSUNDEEP
ruby-on-railsrubybackground-processsidekiq
user984621
user984621

Reputation: 48453

Rails + Sidekiq: How do I avoid a failed job blocking and slowing down the server?

I have a mailer that send emails to company's clients about new products. The emails are sent out to thousands email addresses every day. Here's the code I use for it:

CRON job schedule:

every 1.day, at: local('7:00am') do
  rake "mailer:daily_product_mailer", environment: 'production' 
end

rake task:

namespace :mailer do  desc "Daily product mailer"
  task :daily_product_mailer => [:environment] do                                            
    DailyProductMailerJob.perform_later
  end
end

Job task:

class DailyProductMailerJob < ApplicationJob
  def perform
    ... preparing data...
    DailyProductMailer.send_products.deliver_now
  end
end

Mailer:

class DailyProductMailer < ApplicationMailer
  ...
  def send_products
    ...
    clients.each do |client|
      @email = all_emails_str(client)

      yield client, nil; next if @email.blank?

      begin
        @token = unsubscribe_token

        Retryable.retryable( tries: 3, sleep: 30, on: [Net::OpenTimeout, Net::SMTPAuthenticationError, Net::SMTPServerBusy]) do
          mail(to: @email, subject: "subject", from: [email protected]).deliver
        end

        yield client, @token
      rescue Net::SMTPSyntaxError => e
        error_msg = "Product mailer sending failed on #{Time.now} with: #{e.message}. e.inspect: #{e.inspect}"
        InfonMailer.test_msg('[email protected]', 'Product Mailer - error', error_msg).deliver_now
        logger.warn error_msg
        yield carrier, @token
        next
      end
    end
  end
  ...
end

What happens here is that every morning is run a rake task (in the background) that calls a method that picks all clients from my database and sends out emails about the latest products.

Since a few days ago, I experience a problem when sending emails to certain email addresses fails, which causes significant performance issues of the whole server - everything in the Rails app is super slow, so I need to log in to the AWS EC2 instance, kill the Sidekiq process, then start it again. And next day the same procedure...

I am trying to identify the problem and I thought that the issue is here:

Retryable.retryable( tries: 3, sleep: 30, on: [Net::OpenTimeout, Net::SMTPAuthenticationError, Net::SMTPServerBusy]) do
  mail(to: @email, subject: "subject", from: [email protected]).deliver
end

This block should try to execute the job that has failed up to 3 times - so I tried to remove it, but it didn't help - the job still stays stuck somewhere (and keeps slowing down the server).

What is the reason of Sidekiq blocking the server here? Is there a way to set Sidekiq to continue sending out emails to the new client email addresses if in the previous iteration occurred any error?

Upvotes: 0

Views: 943

Answers (1)

Lam Phan
Lam Phan

Reputation: 3811

base on the document : https://github.com/mperham/sidekiq/wiki/Error-Handling

if the job failed then it will be retried 25 times (about 21 days) so i guess you put a-lot-of failed jobs every day then the retry queues was the cause of your server be slow.

so I tried to remove it

how did you try to remove it when you didn't log job_id ?

base on document : https://github.com/mperham/sidekiq/wiki/API

i think you can try selecting all jobs of a certain type and deleting them from the retry queue :

query = Sidekiq::RetrySet.new
query.select do |job|
  job.klass == 'Sidekiq::Extensions::DelayedClass' &&
    # For Sidekiq::Extensions (e.g., Foo.delay.bar(*args)),
    # the context is serialized to YAML, and must
    # be deserialized to get to the original args
    ((klass, method, args) = YAML.load(job.args[0])) &&
    klass == User &&
    method == :setup_new_subscriber
end.map(&:delete)

# a better approach using 6.0's `scan` to pre-filter
query.scan("setup_new_subscriber").select do |job|
  job.klass == 'Sidekiq::Extensions::DelayedClass' &&
    # For Sidekiq::Extensions (e.g., Foo.delay.bar(*args)),
    # the context is serialized to YAML, and must
    # be deserialized to get to the original args
    ((klass, method, args) = YAML.load(job.args[0])) &&
    klass == User &&
    method == :setup_new_subscriber
end.map(&:delete)

Upvotes: 0

Related Questions