hang issue when connecting to aws using python boto3

Question

I am trying to connect to aws s3 using following steps. But the command s3.meta.client.head_bucket hanged almost 30min. Is there anyway to know the reason for hang or do we keep any checks before connecting to aws s3 to make sure the connection is proper or can we set the timeout?

import boto3
import botocore
boto3.setup_default_session(profile_name='aws_profile')
s3=boto3.resource('s3')
s3.meta.client.head_bucket(Bucket='pha-bucket')
Traceback (most recent call last):
File "", line 1, in
File "/opt/freeware/lib/python2.7/site-packages/botocore/client.py", line 253, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/opt/freeware/lib/python2.7/site-packages/botocore/client.py", line 531, in _make_api_call
operation_model, request_dict)
File "/opt/freeware/lib/python2.7/site-packages/botocore/endpoint.py", line 141, in make_request
return self._send_request(request_dict, operation_model)
File "/opt/freeware/lib/python2.7/site-packages/botocore/endpoint.py", line 170, in _send_request
success_response, exception):
File "/opt/freeware/lib/python2.7/site-packages/botocore/endpoint.py", line 249, in _needs_retry
caught_exception=caught_exception, request_dict=request_dict)
File "/opt/freeware/lib/python2.7/site-packages/botocore/hooks.py", line 227, in emit
return self._emit(event_name, kwargs)
File "/opt/freeware/lib/python2.7/site-packages/botocore/hooks.py", line 210, in _emit
response = handler(**kwargs)
File "/opt/freeware/lib/python2.7/site-packages/botocore/retryhandler.py", line 183, in call
if self._checker(attempts, response, caught_exception):
File "/opt/freeware/lib/python2.7/site-packages/botocore/retryhandler.py", line 251, in call
caught_exception)
File "/opt/freeware/lib/python2.7/site-packages/botocore/retryhandler.py", line 277, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/opt/freeware/lib/python2.7/site-packages/botocore/retryhandler.py", line 317, in call
caught_exception)
File "/opt/freeware/lib/python2.7/site-packages/botocore/retryhandler.py", line 223, in call
attempt_number, caught_exception)
File "/opt/freeware/lib/python2.7/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://s3.ap-south-1.amazonaws.com/pha-bucket"

logs from logging module:
02-12-2021T04:30:35|connectionpool.py[735]|INFO:Starting new HTTPS connection (1): s3.ap-south-1.amazonaws.com
02-12-2021T04:35:55|connectionpool.py[735]|INFO:Starting new HTTPS connection (2): s3.ap-south-1.amazonaws.com
02-12-2021T04:41:17|connectionpool.py[735]|INFO:Starting new HTTPS connection (3): s3.ap-south-1.amazonaws.com
02-12-2021T04:46:40|connectionpool.py[735]|INFO:Starting new HTTPS connection (4): s3.ap-south-1.amazonaws.com
02-12-2021T04:52:07|connectionpool.py[735]|INFO:Starting new HTTPS connection (5): s3.ap-south-1.amazonaws.com

Answer 1

Another way to access aws resources, is using a session.

A session manages state about a particular configuration. By default, a session is created for you when needed. However, it's possible and recommended that in some scenarios you maintain your own session. Sessions typically store the following:

• Credentials
• AWS Region
• Other configurations related to your profile (for example assume a role with more permissions or less permissions depending on use case)

Here is an example:

import boto3

session = boto3.Session(
aws_access_key_id='AWS_ACCESS_KEY_ID', #set manually or by envvar
aws_secret_access_key='AWS_SECRET_ACCESS_KEY', #set manually or by envvar
)
s3 = session.resource('s3')
bucket = s3.Bucket('my-personal-test')

for my_bucket_object in bucket.objects.all():
    print(my_bucket_object)

Answer 2

I just tried the same code as you have shared, it just worked.:

In [1]: import boto3

In [2]: import botocore

In [3]: boto3.setup_default_session(profile_name='myprofile')

In [4]: s3=boto3.resource('s3')
   ...: s3.meta.client.head_bucket(Bucket='zappaadsaziuis7v4f')
Out[4]:
{'ResponseMetadata': {'RequestId': '0CCCB0E3D17D9948',
  'HostId': 'Eu96QWMyG+Ip9XedndlUBemQ7eE9Ps9Lzl1q2NOqi3fbcADEbdo=',
  'HTTPStatusCode': 200,
  'HTTPHeaders': {'x-amz-id-2': 'Eu96QWMyG+Ip9XedndlUBemQ7eE9fbcADEbdo=',
   'x-amz-request-id': '0CCCB0E3D17D9948',
   'date': 'Tue, 16 Feb 2021 20:46:57 GMT',
   'x-amz-bucket-region': 'eu-central-1',
   'content-type': 'application/xml',
   'server': 'AmazonS3'},
  'RetryAttempts': 1}}

In [7]: s3.meta.client.head_bucket(Bucket='mytestbucketzpl')
Out[7]:
{'ResponseMetadata': {'RequestId': '4F10A0EBE7577A78',
  'HostId': 'vfA4aUVnbcrO1glIGe7rm9WMyvwg7b5ZT1NTrq',
  'HTTPStatusCode': 200,
  'HTTPHeaders': {'x-amz-id-2': '',
   'x-amz-request-id': '',
   'date': 'Tue, 16 Feb 2021 20:36:00 GMT',
   'x-amz-bucket-region': 'ap-south-1',},
  'RetryAttempts': 0}}

I would suggest taking a look at this How can I troubleshoot the "Could not connect to the endpoint URL" error when I run the sync command on my Amazon S3 bucket?

Or try an alternate approach as described in boot3 docs:

Accessing a bucket

# Boto3
import botocore
bucket = s3.Bucket('mybucket')
exists = True
try:
    s3.meta.client.head_bucket(Bucket='mybucket')
except botocore.exceptions.ClientError as e:
    # If a client error is thrown, then check that it was a 404 error.
    # If it was a 404 error, then the bucket does not exist.
    error_code = e.response['Error']['Code']
    if error_code == '404':
        exists = False

Answer 3

You need to mention the access_key and secret_key when connecting to a resources, if you haven't configured aws using the aws-configure command previously.

s3=boto3.resource('s3', access_key='xyz', secret_key='scz')