`block (2 levels) in load' exception when running capistrano

Question

Yesterday an unwelcomed visitor was mucking around my server. I since rebooted ubuntu and changed passwords. Now I try to deploy my rails application and I am getting an error.

Here is the important part:

 ** [out :: 172.27.12.xxx] /usr/local/rvm/rubies/ruby-1.9.3-p392/bin/ruby /usr/local/rvm/gems/ruby-1.9.3-p392/bin/rake assets:precompile:all RAILS_ENV=production RAILS_GROUPS=assets
 ** [out :: 172.27.12.xxx] /usr/local/rvm/rubies/ruby-1.9.3-p327/bin/ruby /usr/local/rvm/gems/ruby-1.9.3-p327/bin/rake assets:precompile:all RAILS_ENV=production RAILS_GROUPS=assets
 ** [out :: 172.27.12.xxx] /usr/local/rvm/rubies/ruby-1.9.3-p327/bin/ruby /usr/local/rvm/gems/ruby-1.9.3-p327@global/bin/rake assets:precompile:all RAILS_ENV=production RAILS_GROUPS=assets
*** [deploy:update_code] rolling back
  * executing "rm -rf /home/guarddog/public_html/guarddog.com/releases/20131220150214; true"
    servers: ["172.27.12.xxx", "172.27.12.xxx", "172.27.12.xxx"]
 ** [deploy:update_code] exception while rolling back: IOError, closed stream

This is stack trace of error:

Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/net-ssh-2.6.2/lib/net/ssh/buffered_io.rb:65:in `recv': Operation timed out - recvfrom(2) (Errno::ETIMEDOUT)
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/net-ssh-2.6.2/lib/net/ssh/buffered_io.rb:65:in `fill'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/net-ssh-2.6.2/lib/net/ssh/connection/session.rb:228:in `block in postprocess'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/net-ssh-2.6.2/lib/net/ssh/connection/session.rb:224:in `each'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/net-ssh-2.6.2/lib/net/ssh/connection/session.rb:224:in `postprocess'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/processable.rb:31:in `block in process_iteration'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/processable.rb:43:in `block in ensure_each_session'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/processable.rb:41:in `each'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/processable.rb:41:in `ensure_each_session'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/processable.rb:29:in `process_iteration'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/command.rb:165:in `block (2 levels) in process!'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/command.rb:164:in `loop'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/command.rb:164:in `block in process!'
    from /Users/guarddog/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/command.rb:163:in `process!'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/command.rb:134:in `process'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/actions/invocation.rb:185:in `block in run_tree'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/connections.rb:198:in `block in execute_on_servers'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/connections.rb:186:in `each'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/connections.rb:186:in `each_slice'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/connections.rb:186:in `execute_on_servers'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/actions/invocation.rb:183:in `run_tree'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/actions/invocation.rb:155:in `run'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/namespaces.rb:191:in `method_missing'
    from ./config/deploy.rb:77:in `block (2 levels) in load'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/execution.rb:138:in `instance_eval'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/execution.rb:138:in `invoke_task_directly'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/callbacks.rb:25:in `invoke_task_directly_with_callbacks'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/execution.rb:89:in `execute_task'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/execution.rb:101:in `find_and_execute_task'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/callback.rb:38:in `call'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/callbacks.rb:141:in `block in trigger'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/callbacks.rb:141:in `each'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/callbacks.rb:141:in `trigger'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/callbacks.rb:27:in `invoke_task_directly_with_callbacks'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/execution.rb:89:in `execute_task'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/namespaces.rb:191:in `method_missing'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/namespaces.rb:110:in `block in define_task'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/recipes/deploy.rb:228:in `block (3 levels) in load'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/execution.rb:56:in `transaction'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/namespaces.rb:191:in `method_missing'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/recipes/deploy.rb:227:in `block (2 levels) in load'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/execution.rb:138:in `instance_eval'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/execution.rb:138:in `invoke_task_directly'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/callbacks.rb:25:in `invoke_task_directly_with_callbacks'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/execution.rb:89:in `execute_task'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/namespaces.rb:191:in `method_missing'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/namespaces.rb:110:in `block in define_task'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/recipes/deploy.rb:195:in `block (2 levels) in load'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/execution.rb:138:in `instance_eval'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/execution.rb:138:in `invoke_task_directly'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/callbacks.rb:25:in `invoke_task_directly_with_callbacks'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/execution.rb:89:in `execute_task'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/configuration/execution.rb:101:in `find_and_execute_task'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/cli/execute.rb:46:in `block in execute_requested_actions'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/cli/execute.rb:45:in `each'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/cli/execute.rb:45:in `execute_requested_actions'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/cli/help.rb:19:in `execute_requested_actions_with_help'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/cli/execute.rb:34:in `execute!'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/lib/capistrano/cli/execute.rb:14:in `execute'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/gems/capistrano-2.13.5/bin/cap:4:in `<top (required)>'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/bin/cap:19:in `load'
    from /Users/guarddog/.rvm/gems/ruby-1.9.3-p0/bin/cap:19:in `<main>'

I think the notable part is this line:

from ./config/deploy.rb:77:in `block (2 levels) in load'

That's actual code in my deploy.rb file:

task :precompile, :role => :app do
  run "cd #{release_path}/ && RAILS_ENV=production bundle exec rake assets:precompile"
end

which is executed because of this line in my deploy.rb:

after "deploy:update_code", "deploy:precompile"

Note the deploys were working fine until the incident happened yesterday. He seemed to clone the git repository and copy it somewhere. But he didn't seem to delete any files.

Any idea why deploy:precompile gives the error?

Answer 1

It sounds like your remote codebase may have been compromised. I suggest using

cap deploy:check
cap deploy:cleanup
cap deploy:cold

to redeploy a fresh copy of your original codebase.

This will not modify your remote database, but it will restart the remote httpd server so, as with any deploy, you should wait until a suitable maintenance window to try it.