CodeDeploy fails at DownloadBundle saying Broken pipe @ io_write - <STDERR>

Question

I am using CloudFormation, CodePipeline, CodeDeploy, & CodeCommit to deploy a Rails app to an EC2 instance.

Once the stack is built, CodePipeline kicks off and the first deploy succeeds without issue, but each deploy after that fails at the DownloadBundle step. If I restart the codedeploy-agent on the EC2 instance then the next deploy will succeed, but the following deploys will fail as before.

From the AWS Dashboard, CodeDeploy complains that "too few healthy instances are available" (Error code: HEALTH_CONSTRAINTS)

Under the "View Events" link of CodeDeploy it shows:

Error Code:   UnknownError
Script Name
Message:      Broken pipe @ io_write - <STDERR>
Log Tail

Here is the output from the codedeploy-agent.log:

2017-02-23 21:18:33 INFO  [codedeploy-agent(28647)]: [Aws::CodeDeployCommand::Client 200 0.041592 0 retries] put_host_command_complete(command_status:"Failed",diagnostics:{format:"JSON",payload:"{\"error_code\":5,\"script_name\":\"\",\"message\":\"Broken pipe @ io_write - <STDERR>\",\"log\":\"\"}"},host_command_identifier:" . . . . ")

2017-02-23 21:18:33 ERROR [codedeploy-agent(28647)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Error during perform: Errno::EPIPE - Broken pipe @ io_write - <STDERR> - /usr/local/lib/ruby/gems/2.2.0/gems/rubyzip-1.2.1/lib/zip/entry.rb:573:in `write' /usr/local/lib/ruby/gems/2.2.0/gems/rubyzip-1.2.1/lib/zip/entry.rb:573:in `warn' /usr/local/lib/ruby/gems/2.2.0/gems/rubyzip-1.2.1/lib/zip/entry.rb:573:in `rescue in set_time' /usr/local/lib/ruby/gems/2.2.0/gems/rubyzip-1.2.1/lib/zip/entry.rb:571:in `set_time' /usr/local/lib/ruby/gems/2.2.0/gems/rubyzip-1.2.1/lib/zip/entry.rb:365:in `read_c_dir_entry' /usr/local/lib/ruby/gems/2.2.0/gems/rubyzip-1.2.1/lib/zip/entry.rb:191:in `read_c_dir_entry' /usr/local/lib/ruby/gems/2.2.0/gems/rubyzip-1.2.1/lib/zip/central_directory.rb:127:in `block in read_central_directory_entries' /usr/local/lib/ruby/gems/2.2.0/gems/rubyzip-1.2.1/lib/zip/central_directory.rb:126:in `times' /usr/local/lib/ruby/gems/2.2.0/gems/rubyzip-1.2.1/lib/zip/central_directory.rb:126:in `read_central_directory_entries' /usr/local/lib/ruby/gems/2.2.0/gems/rubyzip-1.2.1/lib/zip/central_directory.rb:138:in `read_from_stream' /usr/local/lib/ruby/gems/2.2.0/gems/rubyzip-1.2.1/lib/zip/file.rb:77:in `block in initialize' /usr/local/lib/ruby/gems/2.2.0/gems/rubyzip-1.2.1/lib/zip/file.rb:76:in `open' /usr/local/lib/ruby/gems/2.2.0/gems/rubyzip-1.2.1/lib/zip/file.rb:76:in `initialize' /usr/local/lib/ruby/gems/2.2.0/gems/rubyzip-1.2.1/lib/zip/file.rb:98:in `new' /usr/local/lib/ruby/gems/2.2.0/gems/rubyzip-1.2.1/lib/zip/file.rb:98:in `open' /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:317:in `unpack_bundle' /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:98:in `block in <class:CommandExecutor>' /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:62:in `execute_command' /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_poller.rb:132:in `process_command' /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_poller.rb:65:in `perform' /opt/codedeploy-agent/lib/instance_agent/agent/base.rb:28:in `run' /opt/codedeploy-agent/lib/instance_agent/runner/child.rb:38:in `block in run' /opt/codedeploy-agent/lib/instance_agent/runner/child.rb:55:in `with_error_handling' /opt/codedeploy-agent/lib/instance_agent/runner/child.rb:37:in `run' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:70:in `block in run_with_error_handling' /opt/codedeploy-agent/lib/instance_agent/runner/child.rb:55:in `with_error_handling' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:69:in `run_with_error_handling' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:33:in `block in start' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:22:in `loop' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:22:in `start' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:206:in `block in spawn_child' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:204:in `fork' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:204:in `spawn_child' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:196:in `block in spawn_children' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:195:in `times' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:195:in `spawn_children' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:134:in `start' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:37:in `block in start' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:36:in `fork' /opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:36:in `start' /opt/codedeploy-agent/bin/../lib/codedeploy-agent.rb:41:in `block (2 levels) in <main>' /opt/codedeploy-agent/vendor/gems/gli-2.5.6/lib/gli/command_support.rb:130:in `call' /opt/codedeploy-agent/vendor/gems/gli-2.5.6/lib/gli/command_support.rb:130:in `execute' /opt/codedeploy-agent/vendor/gems/gli-2.5.6/lib/gli/app_support.rb:262:in `block in call_command' /opt/codedeploy-agent/vendor/gems/gli-2.5.6/lib/gli/app_support.rb:275:in `call' /opt/codedeploy-agent/vendor/gems/gli-2.5.6/lib/gli/app_support.rb:275:in `call_command' /opt/codedeploy-agent/vendor/gems/gli-2.5.6/lib/gli/app_support.rb:69:in `run' /opt/codedeploy-agent/bin/../lib/codedeploy-agent.rb:88:in `<main>'

I have seen others having similar problems, but not specifically what I'm seeing here.

Some solutions I've come across:

Configure a cron to periodically restart the codedeploy agent. # Restarting the agent is not a practical solution.

Other issues seem to be related to how the code is pulled from S3 or GitHub. # This deploy is using CodeCommit instead of S3 or GitHub.

I can provide more detail if needed.

Answer 1

Solved

It turns out that one of my deploy scripts was generating a ruby error with the CodeDeploy agent (see the codedeploy-agent output in the original question). Once I removed that script from my appspec.yml, the deploy began working again.

This was not very intuitive since the CodeDeploy error was showing up in the DownloadBundle step. Apparently the deploy was progressing past that step and failing on the second deploy script in the AfterInstall step.

I wasted a lot of time chasing what I thought was a permissions issue.