CODEWITHSUNDEEP
ruby-on-railsrubysidekiq
gkalil
gkalil

Reputation: 11

Is it possible a single Sidekiq Process running in multiple cores?

I'm using Sidekiq in an MRI/YARV Ruby 2.4.6. I expected that each Sidekiq process would be limited to one core, according to the MRI GIL limitation. But it seems that the threads are running in different cores, even with just one Sidekiq process started.

Here is an htop screenshot right after running: bundle exec sidekiq -c 3 -L sidekiq.log -r ./worker.rb

https://user-images.githubusercontent.com/4698528/63443969-1c012700-c40c-11e9-85ee-9c4298c98c89.png

*3 cores are being used

I read that MRI can make use of OS pthreads to run a process in different cores. Is this what is happening? Does this changes common strategies like running multiple Sidekiq processes to use all the processor capacity?

Upvotes: 1

Views: 674

Answers (1)

Chris Heald
Chris Heald

Reputation: 62698

I suspect that this is reporting issue caused by some quirk of htop + your CPU scheduler; MRI should only be able to use 1 core's worth of CPU due to the GIL, but the CPU scheduler need not necessarily constrain that to a single logical core; the execution may be spread across multiple cores as the scheduler sees fit.

You can observe this trivially:

ruby -e "while(1) do end" &; PID=$!; watch -n 1 ps --pid $PID -o psr; kill $PID

As you watch this, you'll see the CPU core the process is running on change over time, though the usage remains at a constant 100%.

To see this in htop:

ruby -e "4.times.map{Thread.new{while(1) do end}}.map(&:join)" &; PID=$!; htop -p $PID ; kill $PID

You'll notice that in htop, even though your Sidekiq processes are reporting using 50% of a CPU, htop's overall CPU display is showing those cores at only ~30% utilization (at numbers that sum to ~100%!)

Upvotes: 4

Related Questions