Ruby on Rails Sidekiq Worker doesn't complete process, but no error is shown - Processing 30,000+ records in batches

Question

I’m facing an issue with a Sidekiq worker that processes a large number of records in batches, but it’s not completing the process correctly. The worker starts executing, but never reaches the "finished" log, and although there is no explicit error, the process doesn’t complete as expected.

Here’s a summary of what’s happening:

• The worker starts processing HeroProfile (about 30,000 records).
• In the logs, I can see that about 800 records are processed per execution every day.
• The "HeroIntake::ReproveWorker finished" log never appears, which suggests the worker isn't completing the job.
• No explicit errors are seen in the logs, and nothing is captured in Honeybadger.
• When processing, I can see it logging that it's processing records, but it never finishes.

Code:

 # frozen_string_literal: true

 module HeroIntake
  class ReproveWorker
    include Sidekiq::Worker

    sidekiq_options queue: 'low', retry: 1

    def perform
      Rails.logger.info "HeroIntake::ReproveWorker started for: #{HeroProfile.analyze.count} at: #{Time.current}"

      HeroProfile.analyze.where(analyzed_at: nil).find_each(batch_size: 10) do |hero_profile|
        begin
          Rails.logger.info "HeroIntake::ReproveWorker Processing hero profile with ID: #{hero_profile.inspect}"
          Hero::Profile::Reprove.call(hero_profile)
          hero_profile.update_column(:analyzed_at, Time.current)
        rescue StandardError => e
          Honeybadger.notify(error_class: e, error_message: 'HeroIntake::ReproveWorker Error while processing hero rejection')
          Rails.logger.error "HeroIntake::ReproveWorker Error processing hero profile: #{e.message}  ||  #{hero_profile.inspect}"
        end
      end

      Rails.logger.info "HeroIntake::ReproveWorker finished at: #{Time.current}"
    rescue StandardError => e
      Honeybadger.notify(error_class: e, error_message: 'HeroIntake::ReproveWorker Error while processing the worker')
      Rails.logger.error "HeroIntake::ReproveWorker Error: #{e.message}"
    end
  end
end

Observed Behavior:

• The worker starts, but doesn’t finish: The "HeroIntake::ReproveWorker finished" log never appears.
• The worker processes about 800 records per execution, but never exceeds that number, even though there are 30,000 records to process.
• No explicit errors are shown in the logs.

What I’ve Tried:

• Detailed Logs: I increased the logging to capture details about each step, but still didn’t find any obvious errors.
• Timeouts: The worker doesn’t throw a timeout or any other explicit exception.
• Using find_each(batch_size: 10): The worker is processing in batches to prevent memory overload, but the issue persists.
• Sidekiq retry settings: I tested with retry: 0 in sidekiq_options, but that didn’t resolve the issue.
• Time Zone Configurations: I checked the system’s time zone settings, and everything seems correct.

Questions:

• What could be causing the worker to fail to complete processing, even without explicit errors?
• Is there an issue with using find_each or another technique for processing large datasets that might be affecting performance?
• Should I review my Sidekiq configuration to optimize batch processing of records?
• How can I ensure the worker completes, without being interrupted or blocked?

Ruby version: 2.3.8
Rails: gem 'rails', '4.2.11'
Environment: Production
sidekiq gem:

• gem 'sidekiq'
• gem 'sidekiq-failures'
• gem 'sidekiq-scheduler', '~> 3.0', '>= 3.0.1'