CODEWITHSUNDEEP
rubyamazon-web-servicesamazon-elastic-beanstalkamazon-cloudwatch
micah
micah

Reputation: 8106

ElasticBeanstalk Execution Expired When Configuring LogStreaming

I'm getting an error I've never encountered before. I have a multi-region stackset that builds an ElasticBeanstalk application in two regions (us-east-1 and eu-west-1). The configurations are exactly the same except for VPC specific configurations like subnets.

After rebuilding both eb environments, my us-east-1 environment was unable to successfully deploy.

The logs show that one of the hooks are not completing. That hook is-

/opt/elasticbeanstalk/addons/logstreaming/hooks/config/10-config.sh

The contents of the hook look like-

#!/bin/bash -e
#==============================================================================
# Copyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Amazon Software License (the "License"). You may not use
# this file except in compliance with the License. A copy of the License is
# located at
#
#       http://aws.amazon.com/asl/
#
# or in the "license" file accompanying this file. This file is distributed on
# an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, express or
# implied. See the License for the specific language governing permissions
# and limitations under the License.
#==============================================================================

source /opt/elasticbeanstalk/lib/ruby/profile.sh

BASEDIR=$(dirname $0)
ruby $BASEDIR/../logstreaming_config.rb

There isn't much going on in here other than setting up some common ruby env vars in profile.sh and then calling out to logstreaming_config.rb. Following that file, the contents are-

#!/opt/elasticbeanstalk/lib/ruby/bin/ruby

#==============================================================================
# Copyright 2014 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Amazon Software License (the "License"). You may not use
# this file except in compliance with the License. A copy of the License is
# located at
#
#       https://aws.amazon.com/asl/
#
# or in the "license" file accompanying this file. This file is distributed on
# an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, express or
# implied. See the License for the specific language governing permissions
# and limitations under the License.
#==============================================================================
require 'elasticbeanstalk/log-streaming-config-manager'

if __FILE__ == $0
  begin
    log_streaming_config_manager = ElasticBeanstalk::LogStreamingConfigManager.new
    log_streaming_config_manager.config_cwl_agent
  rescue Exception => ex
    puts ex.message
    exit(1)
  end
end

The troubling line here is the one where config_cwl_agent is called. The code for this can be found at /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/beanstalk-core-2.12/lib/elasticbeanstalk/log-streaming-config-manager.rb on the aws box. Which is a largish file so I won't add it here.

Adding logging throughout, I narrow it down to-

cloudwatch_logs.create_log_stream({
  log_group_name: "#{log_group_name}",
  log_stream_name: "#{instance_id}"
})

I haven't drilled any deeper. It managed to create the logstreams themselves at some point, but it never gets over this line since. The only exception that's thrown is one that reads-

execution expired

After ~60s.

I could try rebuilding the environment with a new log_group_name and see if it's resolved. Fortunately this is only happening in a dev environment, so it would be nice to know how to resolve this issue in its current state if I saw this happening in a production environment. Spinning up a new stackset with entirely new eb environments over multiple regions takes a considerable amount of time.

Upvotes: 2

Views: 232

Answers (1)

micah
micah

Reputation: 8106

We found the issue was with VPC Endpoints and the use of Private DNS. We disable Private DNS on the in-region logs endpoint and resolved the issue.

Upvotes: 1

Related Questions