CODEWITHSUNDEEP
ruby-on-railsherokusidekiq
Nick ONeill
Nick ONeill

Reputation: 7381

How to get Sidekiq workers running on Heroku

I've set up Sidekiq with my Rails project. It's running on Heroku with Unicorn. I've gone through all the configuration steps including setting the proper REDISTOGO_URL (as this question references), I've added the following to my after_fork in unicorn.rb:

after_fork do |server,worker|
    if defined?(ActiveRecord::Base)
        ActiveRecord::Base.establish_connection
        Rails.logger.info('Connected to ActiveRecord')
    end

    Sidekiq.configure_client do |config|
        config.redis = { :size => 1 }
    end
end

My Procfile is as follows:

web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
worker: bundle exec sidekiq

Right now I call my worker to perform_async and it adds the task to the queue. In fact in my Sidekiq web interface it says there are 7 items in the queue and it has all of the data there. Yet there are no workers processing the queue and for the life of me, I can't figure out why. If I run 

heroku ps

I get the following output:

=== web: `bundle exec unicorn -p $PORT -c ./config/unicorn.rb`
web.1: up 2012/12/09 08:04:24 (~ 9m ago)

=== worker: `bundle exec sidekiq`
worker.1: up 2012/12/09 08:04:08 (~ 10m ago)

Anybody have any idea what's going on here?

Update

Here's the code for my worker class. Yes, I'm aware that the Oj gem has some issues potentially with sidekiq, but figured I'd give it a shot first. I'm not getting any error messages at this point (the workers don't even run).

require 'addressable/uri'
class DatasiftInteractionsWorker
include Sidekiq::Worker
sidekiq_options queue: "tweets"

def perform( stream_id , interactions )
    interactions = Oj.load(interactions)
    interactions.each{ |interaction|
        if interaction['interaction']['type'] == 'twitter'
            url = interaction['links']['normalized_url'] unless interaction['links']['normalized_url'][0].nil?
            url = interaction['links']['url'] if interaction['links']['normalized_url'][0].nil?
            begin
                puts interaction['links'] if url[0].nil?
                next if url[0].nil?
                host = Addressable::URI.parse(url[0]).host
                host = host.gsub(/^www\.(.*)$/,'\1')
                date_str = Time.now.strftime('%Y%m%d')
                REDIS.pipelined do
                    # Add domain to Redis domains Set
                    REDIS.sadd date_str , host
                    # INCR Redis host
                    REDIS.incr( host + date_str )
                end
            rescue
                puts "ERROR: Could not store the following links: " + interaction['links'].to_s
            end
        end
    }
end
end

Upvotes: 3

Views: 8831

Answers (2)

Billy Coover
Billy Coover

Reputation: 3837

My preference is to create a /config/sidekiq.yml file and then use worker: bundle exec sidekiq -C config/sidekiq.yml in your Procfile.

Here's an example sidekiq.yml file

Upvotes: 6

Nick ONeill
Nick ONeill

Reputation: 7381

Figured out that if you're using a custom queue, you need to make Sidekiq aware of that queue in the Procfile, as follows:

worker: bundle exec sidekiq -q tweets, 1 -q default

I'm not quite sure why this is the case since sidekiq is aware of all queues. I'll post this as an issue on the Sidekiq project.

Upvotes: 4

Related Questions