Boto3 timeout troubleshooting

Question

I am trying to troubleshoot a situation.

I am initiating a Boto3 client like this:

s3_client = boto3.client('s3')

Then I am iterating over a number of files and uploading them using:

s3_client.upload_file()

My problem is that every now and then I see it pausing for 60 seconds and then continuing normally.

I've tried reducing the timeout value and the error I seem to be getting is:

botocore.exceptions.ConnectTimeoutError

or

urllib3.exceptions.ConnectTimeoutError

If I leave it at 60, eventually it succeeds and I don't get any errors.

My question is, this error means that when executing upload_file, it tried to send a put request to the server and the server didn't respond to that request?

When initiating the client, is any connection established that could be lost or that's simply only storing the credentials and it's irrelevant to this issue?

Many thanks.

Update: Adding more detailed logs:

CRITICAL Connect timeout

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 169, in _new_conn
    conn = connection.create_connection(
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
TimeoutError: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/.local/lib/python3.10/site-packages/botocore/httpsession.py", line 439, in send
    urllib_response = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 507, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1012, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 353, in connect
    conn = self._new_conn()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 174, in _new_conn
    raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError

During handling of the above exception, another exception occurred:

  File "/home/user/.local/lib/python3.10/site-packages/boto3/s3/inject.py", line 143, in upload_file
    return transfer.upload_file(
  File "/home/user/.local/lib/python3.10/site-packages/boto3/s3/transfer.py", line 288, in upload_file
    future.result()
  File "/home/user/.local/lib/python3.10/site-packages/s3transfer/futures.py", line 103, in result
    return self._coordinator.result()
  File "/home/user/.local/lib/python3.10/site-packages/s3transfer/futures.py", line 266, in result
    raise self._exception
  File "/home/user/.local/lib/python3.10/site-packages/s3transfer/tasks.py", line 139, in __call__
    return self._execute_main(kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/s3transfer/tasks.py", line 162, in _execute_main
    return_value = self._main(**kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/s3transfer/upload.py", line 758, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/home/user/.local/lib/python3.10/site-packages/botocore/client.py", line 415, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/botocore/client.py", line 731, in _make_api_call
    http, parsed_response = self._make_request(
  File "/home/user/.local/lib/python3.10/site-packages/botocore/client.py", line 751, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/home/user/.local/lib/python3.10/site-packages/botocore/endpoint.py", line 107, in make_request
    return self._send_request(request_dict, operation_model)
  File "/home/user/.local/lib/python3.10/site-packages/botocore/endpoint.py", line 183, in _send_request
    while self._needs_retry(attempts, operation_model, request_dict,
  File "/home/user/.local/lib/python3.10/site-packages/botocore/endpoint.py", line 305, in _needs_retry
    responses = self._event_emitter.emit(
  File "/home/user/.local/lib/python3.10/site-packages/botocore/hooks.py", line 358, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/botocore/hooks.py", line 229, in emit
    return self._emit(event_name, kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/botocore/hooks.py", line 212, in _emit
    response = handler(**kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/botocore/retryhandler.py", line 194, in __call__
    if self._checker(**checker_kwargs):
  File "/home/user/.local/lib/python3.10/site-packages/botocore/retryhandler.py", line 267, in __call__
    should_retry = self._should_retry(attempt_number, response,
  File "/home/user/.local/lib/python3.10/site-packages/botocore/retryhandler.py", line 294, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/home/user/.local/lib/python3.10/site-packages/botocore/retryhandler.py", line 333, in __call__
    checker_response = checker(attempt_number, response,
  File "/home/user/.local/lib/python3.10/site-packages/botocore/retryhandler.py", line 233, in __call__
    return self._check_caught_exception(
  File "/home/user/.local/lib/python3.10/site-packages/botocore/retryhandler.py", line 376, in _check_caught_exception
    raise caught_exception
  File "/home/user/.local/lib/python3.10/site-packages/botocore/endpoint.py", line 249, in _do_get_response
    http_response = self._send(request)
  File "/home/user/.local/lib/python3.10/site-packages/botocore/endpoint.py", line 321, in _send
    return self.http_session.send(request)
  File "/home/user/.local/lib/python3.10/site-packages/botocore/httpsession.py", line 472, in send
    raise ConnectTimeoutError(endpoint_url=request.url, error=e)

Answer 1

The default boto3 retry mode (Legacy) doesn't properly handle a number of errors/exceptions, including RequestThrottled, PriorRequestNotComplete, ConnectionError and BandwidthLimitExceeded. It can manifest as very intermittent problems with long delays and repeated connection failures. You'll likely be better off using the Standard retry mode instead, which also has an exponential backoff.

import boto3
from botocore.client import Config as BotoConfig

TIMEOUT = 3
config = BotoConfig(connect_timeout=TIMEOUT, retries={"mode": "standard"})
client = boto3.client("s3", config=config)