Why is aws-sdk-ruby connecting to '169.254.169.254' on initialization?

Question

I installed the gem webmock, which blocks external connections during the test suite. After I installed it, I got an unexpected result when running the test suite.

I'm somewhat confused why the aws-sdk needs to connect to 169.254.169.254 on initialization?

rspec spec/models/concerns/posconcern_spec.rb
[Coveralls] Set up the SimpleCov formatter.
[Coveralls] Using SimpleCov's 'rails' settings.

An error occurred while loading ./spec/models/concerns/posconcern_spec.rb.
Failure/Error: require File.expand_path('../../config/environment', __FILE__)

WebMock::NetConnectNotAllowedError:
  Real HTTP connections are disabled. Unregistered request: GET http://169.254.169.254/latest/meta-data/iam/security-credentials/ with headers {'Accept'=>'*/*', 'Accept-Encoding'=>'gzip;q=1.0,deflate;q=0.6,identity;q=0.3', 'User-Agent'=>'Ruby'}

  You can stub this request with the following snippet:

  stub_request(:get, "http://169.254.169.254/latest/meta-data/iam/security-credentials/").
    with(headers: {'Accept'=>'*/*', 'Accept-Encoding'=>'gzip;q=1.0,deflate;q=0.6,identity;q=0.3', 'User-Agent'=>'Ruby'}).
    to_return(status: 200, body: "", headers: {})

  ============================================================
# /home/andey/.rvm/gems/ruby-2.3.4/gems/webmock-3.1.0/lib/webmock/http_lib_adapters/net_http.rb:114:in `request'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:109:in `http_get'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:90:in `block (2 levels) in get_credentials'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:105:in `open_connection'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:88:in `block in get_credentials'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:121:in `retry_errors'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:87:in `get_credentials'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:73:in `block in refresh'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:121:in `retry_errors'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:72:in `refresh'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/refreshing_credentials.rb:20:in `initialize'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/instance_profile_credentials.rb:51:in `initialize'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/credential_provider_chain.rb:90:in `new'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/credential_provider_chain.rb:90:in `instance_profile_credentials'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/credential_provider_chain.rb:12:in `block in resolve'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/credential_provider_chain.rb:11:in `each'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/credential_provider_chain.rb:11:in `resolve'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/aws-sdk-core/plugins/request_signer.rb:37:in `block in <class:RequestSigner>'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:70:in `call'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:205:in `block in resolve_defaults'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:57:in `each'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:57:in `each'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:204:in `resolve_defaults'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:200:in `value_at'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:189:in `block in resolve'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:189:in `resolve'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:177:in `apply_defaults'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/configuration.rb:150:in `build!'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/base.rb:68:in `build_config'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/base.rb:19:in `initialize'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/aws-sdk-core-2.10.52/lib/seahorse/client/base.rb:105:in `new'
# ./config/initializers/aws.rb:1:in `<top (required)>'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/activesupport-4.2.9/lib/active_support/dependencies.rb:268:in `load'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/activesupport-4.2.9/lib/active_support/dependencies.rb:268:in `block in load'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/activesupport-4.2.9/lib/active_support/dependencies.rb:240:in `load_dependency'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/activesupport-4.2.9/lib/active_support/dependencies.rb:268:in `load'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/engine.rb:652:in `block in load_config_initializer'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/activesupport-4.2.9/lib/active_support/notifications.rb:166:in `instrument'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/engine.rb:651:in `load_config_initializer'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/engine.rb:616:in `block (2 levels) in <class:Engine>'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/engine.rb:615:in `each'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/engine.rb:615:in `block in <class:Engine>'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/initializable.rb:30:in `instance_exec'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/initializable.rb:30:in `run'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/initializable.rb:55:in `block in run_initializers'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/initializable.rb:44:in `each'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/initializable.rb:44:in `tsort_each_child'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/initializable.rb:54:in `run_initializers'
# /home/andey/.rvm/gems/ruby-2.3.4/gems/railties-4.2.9/lib/rails/application.rb:352:in `initialize!'
# ./config/environment.rb:5:in `<top (required)>'
# ./spec/rails_helper.rb:3:in `require'
# ./spec/rails_helper.rb:3:in `<top (required)>'
# ./spec/models/concerns/posconcern_spec.rb:1:in `require'
# ./spec/models/concerns/posconcern_spec.rb:1:in `<top (required)>'

Relevant link to repo:

https://github.com/aws/aws-sdk-ruby/blob/master/gems/aws-sdk-core/lib/aws-sdk-core/instance_profile_credentials.rb#L45

Answer 1

set following ENV variable in CI environment:

AWS_EC2_METADATA_DISABLED=true

Excerpt from the docs:

Disables the use of the Amazon EC2 instance metadata service (IMDS).

If set to true, user credentials or configuration (like the Region) are not requested from IMDS

Answer 2

There are some good answers in this thread on how to stub this. Another alternative (a better one?) is to eliminate the need for thE SDK to check for credentials from the instance meta data service. You can do this by providing credentials. These can be fake. A simple way to do this would be to provide credential via the ENV (e.g. ENV['AWS_ACCESS_TOKEN'] and ENV['SECRET_ACCESS_TOKEN']).

Answer 3

Had a tough time researching on how to stub it. Will share my findings here hope it'll help other people.

Since AWS uses service name to fetch credentials for multiple services such as S3, any IAM Role, SQS etc. But when dealing with IAM roles AWS hits http://169.254.169.254/latest/meta-data/iam/security-credentials/ which return the role name attached onto the machine and using the response of the same it hits the curl http://169.254.169.254/latest/meta-data/iam/security-credentials/#{role}. So we need to stub both of the curls in order to successfully mock the IAM role. Below is the code for the same.

def self.auth(params = {})
  status = params[:status].presence || 200
  url = 'http://169.254.169.254/latest/meta-data/iam/security-credentials/'
  WebMock.stub_request(:get, url)
      .with({headers: request_headers}.compact)
      .to_return(status: status, body: 'SampleRole', headers: {})
end

def self.role_auth(role = 'SampleRole')
  url = "http://169.254.169.254/latest/meta-data/iam/security-credentials/#{role}"
  WebMock.stub_request(:get, url)
      .with({headers: request_headers}.compact)
      .to_return(status: 200, body: sample_response, headers: {})
end

private

def self.request_headers
  {
      'Accept': '*/*',
      'Accept-Encoding': 'gzip;q=1.0,deflate;q=0.6,identity;q=0.3',
      'User-Agent': 'aws-sdk-ruby3/3.21.2'
  }.with_indifferent_access
end

def self.sample_response
  {
      Code: 'Success',
      LastUpdated: '2020-05-08T05:55:23Z',
      Type: 'AWS-HMAC',
      AccessKeyId: Faker::Lorem.characters(10),
      SecretAccessKey: Faker::Lorem.characters(20),
      Token: Faker::Lorem.characters(60),
      Expiration: '2020-05-08T12:00:45Z'
  }.to_json
end

So whenver any rspec uses InstanceCredentials provider you need to stub both of the above defined requests in before block of each of the rspec.

Example:

before do
 ApiStub::IamCred.auth
 ApiStub::IamCred.role_auth
end

Hope it helps. Happy Coding

Answer 4

This is a follow up to @helloV 's answer.

Full documentation on

http://169.254.169.254/latest/meta-data/iam/security-credentials/

An application on the instance retrieves the security credentials provided by the role from the instance metadata item iam/security-credentials/role-name. The application is granted the permissions for the actions and resources that you've defined for the role through the security credentials associated with the role. These security credentials are temporary and we rotate them automatically. We make new credentials available at least five minutes prior to the expiration of the old credentials.

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#instance-metadata-security-credentials

Answer 5

It is trying to connect to the AWS metadata server to get the AWS credentials. All AWS servers should be able to connect to the internal metadata server.

I installed the gem webmock, which blocks external connections during the test suite

How are you blocking it? Firewall? Add a rule to allow traffic to 169.254.169.254 or stub it.

The script is trying to connect to AWS services. For that it needs credentials which can be provided in many ways. One way is to use AWS IAM role and fetch the credentials dynamically from the metadata server (169.254.169.254). Your script is connecting to 169.254.169.254 to fetch the credentials. It can be used to connect to AWS services later.