heroku postgress connection not open error for background job

Question

Error:

ActiveRecord::StatementInvalid: PG::Error: connection not open : UPDATE "delayed_jobs" SET locked_by = null, locked_at = null WHERE (locked_by = '...')

It occurs at rake jobs:work for the app deployed at heroku and using postgres database. Backtrace does not provide me any information about which task caused the failure:

activerecord-3.2.12/lib/active_record/connection_adapters/postgresql_adapter.rb:1153→ async_exec
activerecord-3.2.12/lib/active_record/connection_adapters/postgresql_adapter.rb:1153→ exec_no_cache
activerecord-3.2.12/lib/active_record/connection_adapters/postgresql_adapter.rb:673→ block in exec_delete
activerecord-3.2.12/lib/active_record/connection_adapters/abstract_adapter.rb:280→ block in log
activesupport-3.2.12/lib/active_support/notifications/instrumenter.rb:20→ instrument
activerecord-3.2.12/lib/active_record/connection_adapters/abstract_adapter.rb:275→ log
newrelic_rpm-3.3.3/lib/new_relic/agent/instrumentation/active_record.rb:31→ block in log_with_newrelic_instrumentation
newrelic_rpm-3.3.3/lib/new_relic/agent/method_tracer.rb:242→ trace_execution_scoped
newrelic_rpm-3.3.3/lib/new_relic/agent/instrumentation/active_record.rb:28→ log_with_newrelic_instrumentation
activerecord-3.2.12/lib/active_record/connection_adapters/postgresql_adapter.rb:672→ exec_delete
activerecord-3.2.12/lib/active_record/connection_adapters/abstract/database_statements.rb:96→ update
activerecord-3.2.12/lib/active_record/connection_adapters/abstract/query_cache.rb:14→ update
activerecord-3.2.12/lib/active_record/relation.rb:294→ update_all
activerecord-3.2.12/lib/active_record/relation.rb:278→ update_all
activerecord-3.2.12/lib/active_record/querying.rb:7→ update_all
delayed_job_active_record-0.3.2/lib/delayed/backend/active_record.rb:44→ clear_locks!
delayed_job-3.0.3/lib/delayed/plugins/clear_locks.rb:9→ ensure in block (2 levels) in <class:ClearLocks>
delayed_job-3.0.3/lib/delayed/plugins/clear_locks.rb:9→ block (2 levels) in <class:ClearLocks>
delayed_job-3.0.3/lib/delayed/lifecycle.rb:78→ call
delayed_job-3.0.3/lib/delayed/lifecycle.rb:78→ block (2 levels) in add
delayed_job-3.0.3/lib/delayed/lifecycle.rb:60→ call
delayed_job-3.0.3/lib/delayed/lifecycle.rb:60→ block in initialize
delayed_job-3.0.3/lib/delayed/lifecycle.rb:78→ call
delayed_job-3.0.3/lib/delayed/lifecycle.rb:78→ block in add
delayed_job-3.0.3/lib/delayed/lifecycle.rb:65→ call
delayed_job-3.0.3/lib/delayed/lifecycle.rb:65→ execute
delayed_job-3.0.3/lib/delayed/lifecycle.rb:38→ run_callbacks
delayed_job-3.0.3/lib/delayed/worker.rb:127→ start
delayed_job-3.0.3/lib/delayed/tasks.rb:9→ block (2 levels) in <top (required)>
rake-10.0.3/lib/rake/task.rb:228→ call
rake-10.0.3/lib/rake/task.rb:228→ block in execute
rake-10.0.3/lib/rake/task.rb:223→ each
rake-10.0.3/lib/rake/task.rb:223→ execute
rake-10.0.3/lib/rake/task.rb:166→ block in invoke_with_call_chain
/usr/local/lib/ruby/1.9.1/monitor.rb:201→ mon_synchronize
rake-10.0.3/lib/rake/task.rb:159→ invoke_with_call_chain
rake-10.0.3/lib/rake/task.rb:152→ invoke
rake-10.0.3/lib/rake/application.rb:143→ invoke_task
rake-10.0.3/lib/rake/application.rb:101→ block (2 levels) in top_level
rake-10.0.3/lib/rake/application.rb:101→ each
rake-10.0.3/lib/rake/application.rb:101→ block in top_level
rake-10.0.3/lib/rake/application.rb:110→ run_with_threads
rake-10.0.3/lib/rake/application.rb:95→ top_level
rake-10.0.3/lib/rake/application.rb:73→ block in run
rake-10.0.3/lib/rake/application.rb:160→ standard_exception_handling
rake-10.0.3/lib/rake/application.rb:70→ run
rake-10.0.3/bin/rake:33→ <top (required)>
[PROJECT_ROOT]/vendor/bundle/ruby/1.9.1/bin/rake:19→ load
[PROJECT_ROOT]/vendor/bundle/ruby/1.9.1/bin/rake:19→ <main>

So I guess it occurred during the environment setup. Any help?

Answer 1

It seems that you are using new relic and it seems that there is known issue for this problem look here and here on github

For solution, maybe you could use some alternative solution. Take a look at this question https://stackoverflow.com/questions/4840319/cheaper-alternatives-to-new-relic it might help you. Instead, you could try https://www.scoutapp.com/

Also take a look at this article http://drewblas.com/2008/05/29/comparison-of-rails-monitoring-apps/ and decide what is the best solution for you.

Let me know if you need any additional help

Answer 2

Did you look in postgres logs? https://devcenter.heroku.com/articles/heroku-postgresql#monitoring-logging