CODEWITHSUNDEEP
ruby-on-railsrubyperformanceredissidekiq
Khmtrav
Khmtrav

Reputation: 49

poor Redis performance inside Sidekiq worker

I have a Rails app, that runs 50 concurrent Sidekiq workers, that receive data from external APIs, manipulate it and store it in a Database. In order to prevent to store the same data multiple times, I create unique strings, that I store into Redis. 

r = Redis.current
r.pipelined do
    options.each do |o|
        r.setex(o.generate_index, 1.hour, o.value)
    end
end

Before saving a new object to the Database I check if it's unique string exists in Redis:

r = Redis.current
option_ids = r.pipelined do
    new_options.each do |o|
        r.get(o.generate_index) # check if unique string already exists
    end
end

I also tested without pielining with the same result: Redis GETs/SETs are extremely slow. Some GET-Request take more that one second!

weird thing: While running 50 Sidekiq workers, that had a bad Redis performance, I benchmarked Redis from the shell, with very good results:

r = Redis.current
10000000.times { puts Benchmark.realtime{ 1000.times{|i| r.get(i)} }}

with output like:

0.197379041
0.192059888
0.196165358
0.207617425
0.198095963
0.195844917
0.211404108
0.203188759
0.208119298
0.184018683

No Benchmark took longer than 0.5 second.

I also tried Redis.new instead of Redis.current with the same result.

I can't figure out what's the problem

EDIT: After some more testing and benchmarking I noticed that the benchmarks in my code seemed to block other workers and slow them down much more than expected. Without benchmarking the workers are up to 10 times faster. Furthermore it seems that my vserver just cant handle that many workers. My development laptop can take 100 workers and still performs very fast. Moreover when the load is heavy, database queries seem to be very slow on the ruby side. I benchmarked simple Mongoid lookups like

time = Benchmark.realtime do
    existing_option = Option.find_by(index: o.generate_index)
end

These benchmarks sometimes took longer than one second, but the query did not show up in the mongodb slow log (which means the actual query took less than 100ms on the database). However pure rubycode (without database hits) seems to perform ok. I wonder if it performs better when I implement the HTTP-requests with Typhoeus Hydra instead of sidekiq.

Upvotes: 2

Views: 1786

Answers (1)

Mike Perham
Mike Perham

Reputation: 22208

One thread contacting Redis is very different from 50 threads.

Use Sidekiq's redis connection pool within your workers. It will scale much better.

Sidekiq.redis do |conn|
  conn.pipelined do
    ...
  end
end

Upvotes: 2

Related Questions