Boto3 not directing to the endpoint_url

Question

I'm currently trying to connect to my enterprise s3 URL (which is not amazon web-service) using boto3 and I have the following error.

EndpointConnectionError: Could not connect to the endpoint URL: "https://s3.fr-par.amazonaws.com/my_buket...." which is absolutely not the enpoint given in the code.

s3 = boto3.resource(service_name='s3',
                    aws_access_key_id= 'XXXXXX',
                    aws_secret_access_key='YYYYYYY',
                    endpoint_url= 'https://my_buket.s3.my_region.my_company_enpoint_url')

my_bucket=s3.Bucket(s3_bucket_name)
bucket_list = []
for file in my_bucket.objects.filter(Prefix='boston.csv'):
    bucket_list.append(file.key)

As can be seen in the error image boto3 tries to connect to a amazonaws url, which is not that of my enterprise. Finally I want to indicate that I am able to connect to my enterprise s3 using minIO https://docs.min.io/ which indicate there no errors in the aws_access_key_id, the aws_secret_access_key and endpoint_url I use with boto3.

I have executed the code using a python 3.9 environment (Boto3 version 1.22.1) a anaconda 3.9 environment (Boto3 version 1.22.0) and a jupyter notebook always with same error. The OS is an Ubuntu 20.04.4 LTS virtualized on Oracle VM virtual box.

Answer 1

Since some people seem to have the same problem, I'm posting the solution I found.

For some reason the code in the question still doesn't work for me. Alternatively, I handle pointing to my enterprise's S3 just by first creating a session and creating the resource and client from it. Note that in endpoint_url, no bucket is indicated.

Since there is no bucket in endpoint_url, you have access to all buckets associated with the credential pass, and therefore it is necessary to specify the bucket in the resource and client instances methods.

session = boto3.Session(region_name=my_region)

resource = session.resource('s3', 
                 endpoint_url='https://s3.my_region.my_company_enpoint_url',
                 aws_access_key_id='XXXXXX',
                 aws_secret_access_key='YYYYYY')

client = session.client('s3', 
                 endpoint_url='https://s3.my_region.my_company_enpoint_url',
                 aws_access_key_id='XXXXXX',
                 aws_secret_access_key='YYYYYY')

client.upload_file(path_to_local_file, bucket_name, upload_path, 
                   Callback=call, 
                   ExtraArgs=ExtraArgs)

Answer 2

https://my_buket.s3.my_region.my_company_enpoint_url is not the endpoint. The list of valid S3 endpoints is here. But normally you don't have to specify it explicitly. Boto3 will "know" which endpoint to use for each region.