ActiveRecord::StatementInvalid: Mysql::Error: MySQL server has gone away: (using the Parallel gem)

Question

I am using the Parallel gem with Rails3 and getting issues with mysql threads, even with a simple line like:

Parallel.each(User.all, :in_processes => 1) { |r| puts r.username }

It alternately works and then fails the second time through. Here is the error I get:

ruby-1.8.7-p330 :035 > Parallel.each(User.all, :in_processes => 1) { |r| puts r.username }
ActiveRecord::StatementInvalid: Mysql::Error: MySQL server has gone away: SELECT `users`.* FROM `users`
    from /Users/kimptoc/.rvm/gems/ruby-1.8.7-p330@p-ecom1-rails3/gems/activerecord-3.0.3/lib/active_record/connection_adapters/abstract_adapter.rb:202:in `log'
    from /Users/kimptoc/.rvm/gems/ruby-1.8.7-p330@p-ecom1-rails3/gems/activerecord-3.0.3/lib/active_record/connection_adapters/mysql_adapter.rb:289:in `execute'
    from /Users/kimptoc/.rvm/gems/ruby-1.8.7-p330@p-ecom1-rails3/gems/activerecord-3.0.3/lib/active_record/connection_adapters/mysql_adapter.rb:619:in `select'
    from /Users/kimptoc/.rvm/gems/ruby-1.8.7-p330@p-ecom1-rails3/gems/activerecord-3.0.3/lib/active_record/connection_adapters/abstract/database_statements.rb:7:in `select_all'
    from /Users/kimptoc/.rvm/gems/ruby-1.8.7-p330@p-ecom1-rails3/gems/activerecord-3.0.3/lib/active_record/connection_adapters/abstract/query_cache.rb:56:in `select_all'
    from /Users/kimptoc/.rvm/gems/ruby-1.8.7-p330@p-ecom1-rails3/gems/activerecord-3.0.3/lib/active_record/base.rb:467:in `find_by_sql'
    from /Users/kimptoc/.rvm/gems/ruby-1.8.7-p330@p-ecom1-rails3/gems/activerecord-3.0.3/lib/active_record/relation.rb:64:in `to_a'
    from /Users/kimptoc/.rvm/gems/ruby-1.8.7-p330@p-ecom1-rails3/gems/activerecord-3.0.3/lib/active_record/relation/finder_methods.rb:143:in `all'
    from /Users/kimptoc/.rvm/gems/ruby-1.8.7-p330@p-ecom1-rails3/gems/activerecord-3.0.3/lib/active_record/base.rb:439:in `__send__'
    from /Users/kimptoc/.rvm/gems/ruby-1.8.7-p330@p-ecom1-rails3/gems/activerecord-3.0.3/lib/active_record/base.rb:439:in `all'
    from (irb):35
    from (null):0

If I do the non-parallel version, it works fine:

User.all.each { |r| puts r.username }

I am using the mysql gem, but have tried mysql2 and mysqlplus.

Running on OSX.

I am thinking its an issue with how ActiveRecord and the mysql gem work with threads.

From what I've read, it might be that I need to tweak the mysql settings to make it more concurrency friendly. Although the alternate gems seem to address handling concurrency better.

I have raised this as a query with the gem - https://github.com/grosser/parallel/issues/9#comment_844380 - but this seems like more like a fundamental issue of how to setup mysql with ruby for concurrent access...

So, my question is - is there a definitive configuration for Rails3 and mysql for concurrent DB access?

Thanks, Chris

EDIT

What seems to be working is splitting into 2 queries, one to the get the id's, then looping through the id's in parallel and within the loop, re-accessing the entity by id.

ids = User.all.map { |u| u.id }
Parallel.each(ids, :in_processes => 1) do |uid| 
  ActiveRecord::Base.connection.reconnect!
  r = User.find(uid)
  puts r.username 
end

Answer 1

I was getting a very similar error with the following:

pid = Process.fork
if pid
  Process.detach(pid)
else
  # Perform long task using ActiveRecord
  do_stuff
end

If I hit the server with a request while do_stuff was running, it would kill the task and throw an exception:

ActiveRecord::StatementInvalid (Mysql2::Error: Lost connection to MySQL server during query: ...

Adding Francois' suggestion fixed my problem:

pid = Process.fork
if pid
  Process.detach(pid)
else
  # Perform long task using ActiveRecord
  ActiveRecord::Base.establish_connection
  do_stuff
end

Thanks Francois!

Answer 2

You need to establish connections after forking. That's a "feature" of forking - network connections are in an inconsistent state.

Parallel.each(User.all, :in_processes => 1) do |r|
  ::ActiveRecord::Base.establish_connection
  puts r.username
end