Does concurrency happen even when only one thread is in a thread pool?

Question

I'm using Rails 5 and Ruby 2.4. How can I figure out, or can you tell by looking at the below, whether there are multiple threads running at the same time?

pool = Concurrent::FixedThreadPool.new(1)
promises = links.map do |link|
  Concurrent::Promise.execute(executor: pool) do
    result = process_link(link)
    if result
      if result.kind_of?(Array)
        result.each do |my_obj|
          my_obj.update_attributes({ :a => a })
          records_processed = records_processed + my_obj.matches.count
        end
      else
        records_processed = records_processed + result.matches.count
        result.update_attributes({ :a => a })
      end
    end
  end
end
promises.map(&:wait).map(&:value!)

As I have set my pool to "1," my assumption is that nothing is running concurrently, but I keep getting this error ...

Error during processing: (ActiveRecord::ConnectionTimeoutError) could not obtain a connection from the pool within 5.000 seconds (waited 5.002 seconds); all pooled connections were in use
/Users/nataliab/.rvm/gems/ruby-2.4.0@global/gems/activerecord-5.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:202:in `block in wait_poll'
/Users/nataliab/.rvm/gems/ruby-2.4.0@global/gems/activerecord-5.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:193:in `loop'
/Users/nataliab/.rvm/gems/ruby-2.4.0@global/gems/activerecord-5.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:193:in `wait_poll'
/Users/nataliab/.rvm/gems/ruby-2.4.0@global/gems/activerecord-5.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:154:in `internal_poll'
/Users/nataliab/.rvm/gems/ruby-2.4.0@global/gems/activerecord-5.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:278:in `internal_poll'
/Users/nataliab/.rvm/gems/ruby-2.4.0@global/gems/activerecord-5.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:148:in `block in poll'
/Users/nataliab/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/Users/nataliab/.rvm/gems/ruby-2.4.0@global/gems/activerecord-5.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:158:in `synchronize'
/Users/nataliab/.rvm/gems/ruby-2.4.0@global/gems/activerecord-5.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:148:in `poll'
/Users/nataliab/.rvm/gems/ruby-2.4.0@global/gems/activerecord-5.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:717:in    `acquire_connection'
/Users/nataliab/.rvm/gems/ruby-2.4.0@global/gems/activerecord-5.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:490:in `checkout'
/Users/nataliab/.rvm/gems/ruby-2.4.0@global/gems/activerecord-5.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:364:in `connection'
/Users/nataliab/.rvm/gems/ruby-2.4.0@global/gems/activerecord-5.0.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:883:in `retrieve_connection'
/Users/nataliab/.rvm/gems/ruby-2.4.0@global/gems/activerecord-5.0.1/lib/active_record/connection_handling.rb:128:in `retrieve_connection'

I don't get the above error if I modify my code to run where I am positive there is no concurrency going on ...

links.each do |link|
  result = process_link(link)
  if result
    if result.kind_of?(Array)
      result.each do |race|
        my_obj.update_attributes({ :a => a })
        records_processed = records_processed + my_obj.matches.count
      end
    else
      records_processed = records_processed + result.matches.count
      result.update_attributes({ :a => a })
    end
  end
end

Edit: This is my database configuration for my development environment. Also note that all of this is being run in the rails console.

development:
  adapter: postgresql
  encoding: utf8
  database: sims 
  username: postgres
  password: password
  pool: 5
  timeout: 15000
  host: 127.0.0.1

Answer 1

Your assumption that multiple threads must be running concurrently just because the connection pool is being exhausted is not correct. Just because a connection is still 'checked out' from the connection pool doesn't mean that a query is currently being executed on the checked-out connection in a thread, it simply means that the thread's connection hasn't been checked back in. The thread could be sitting idle but still holding onto a connection from the connection pool as long as it hasn't been explicitly terminated.

Since ActiveRecord Connections are thread-local, you can exhaust the connection pool by running ActiveRecord queries on multiple threads, as you are doing in this case. (Every time Concurrent::FixedThreadPool.new(1) is called, a new thread is created.) Even if you're only running queries on a single thread at a time, by default a connection will still be held open on every thread until they are terminated.

To avoid this, you can either manually check in connections after using them, or ensure your threads are terminated (killed) so that their connections can be recovered (reaped) by the pool.

• To manually check in connections, refer to the ConnectionPool documentation for your options. The easiest way is to wrap your ActiveRecord code in a with_connection block:

  Concurrent::Promise.execute(executor: pool) do
    ActiveRecord::Base.connection_pool.with_connection do
      # update_attributes, etc
    end
  end
  

• To ensure all threads are terminated, call #shutdown followed by #wait_for_termination on the thread pool after you're finished using it:

  values = promises.map(&:value!)
  pool.shutdown
  pool.wait_for_termination
  

Answer 2

You assumption that there is only one thread is incorrect. There are two - the one in the thread pool and the main one that spawned the one in the thread pool.

You might be confused as you made the main thread wait and it shouldn't access the database. That doesn't mean that it still doesn't hold a connection, hence preventing the other thread from acquiring one.

As a rule of thumb your database connection pool should be set to at least number of threads spawned + 1. In this case - 2.

---

Code to easily reproduce:

# migration
class CreateFoos < ActiveRecord::Migration[5.0]
  def change
    create_table :foos do |t|
      t.integer :bar
    end
  end
end

# model
class Foo < ApplicationRecord
end

# rake task
task experiment: :environment do
  Foo.create
  pool = Concurrent::FixedThreadPool.new(1) 
  promise = 
    Concurrent::Promise.execute(executor: pool) do
      Foo.first.update_attributes!(bar: rand(-42..42))
    end
  promise.wait.value!
end

Set pool to 1 in your config/database.yml and run the task. You will get an error. Set it to 2 - it will be just fine.

You can increase the number of threads in the pool and add at least that much promises for processing. You will consistently fail for database connection pool = number of threads in the thread pool and succeed if you add one more in config/database.yml.

Answer 3

Since you are defining the fixed thread pool to have one thread, I would assume that you are not achieving any kind of concurrency. Looking at your error it appears that the one available thread from the pool was busy for too long and caused the connection timeout exception.

When you altered your code implementation so that it did not contain a thread pool, the application was explicitly single threaded without the possibility of a connection timeout due to waiting on threads from a pool. Try increasing the size of your thread pool (perhaps to 3 or 5) and see if you are still getting that same exception.