CODEWITHSUNDEEP
pythonamazon-web-servicesbotoboto3botocore
Vor
Vor

Reputation: 35099

Override S3 endpoint using Boto3 configuration file

OVERVIEW:

I'm trying to override certain variables in boto3 using the configuration file (~/aws/confg). In my use case I want to use fakes3 service and send S3 requests to the localhost.

EXAMPLE:

In boto (not boto3), I can create a config in ~/.boto similar to this one:

[s3]
host = localhost
calling_format = boto.s3.connection.OrdinaryCallingFormat

[Boto]
is_secure = False

And the client can successfully pick up desired changes and instead of sending traffic to the real S3 service, it will send it to the localhost.

>>> import boto
>>> boto.connect_s3()
S3Connection:localhost
>>>

WHAT I TRIED:

I'm trying to achieve a similar result using boto3 library. By looking at the source code I found that I can use ~/aws/config location. I've also found an example config in unittests folder of botocore.

I tried to modify the config to achieve the desired behaviour. But unfortunately, it doesn't work.

Here is the config:

[default]
aws_access_key_id = XXXXXXXXX
aws_secret_access_key = YYYYYYYYYYYYYY
region = us-east-1
is_secure = False
s3 =
    host = localhost

QUESTION:

  1. How to overwrite clients variables using config file?
  2. Where can I find a complete list of allowed variables for the configuration?

Upvotes: 61

Views: 96493

Answers (7)

Q. Qiao
Q. Qiao

Reputation: 827

If you are using https, this is the way to create the s3_client:

s3_client = boto3.client(
    "s3",
    "us-east-1",
    aws_access_key_id="test_id",
    aws_secret_access_key="test_key",
    endpoint_url="https://localhost:7000",
    use_ssl=True,
    verify=False,
)

Upvotes: 6

MichaelM
MichaelM

Reputation: 1109

Setting the AWS endpoint using either a config or environment variables is now supported in 2024!

Setting using the the AWS/Boto config:

[default]
s3 =
  endpoint_url = http://localhost:1234

Setting using an environment variable:

export AWS_ENDPOINT_URL_S3=http://localhost:1234

I've made the examples above specific to S3 like your original question, but these can be easily updated to be global, rather than targeting S3, by removing the s3 = line of the config or the _S3 portion of the environment variable.

Where can I find a complete list of allowed variables for the configuration?

Upvotes: 5

Rachel
Rachel

Reputation: 2894

I've just submitted a PR to boto3 to add an env var to allow you to override the endpoint_url if you need to use a dependent module which uses boto3 (in which case you may not be able to modify the call to boto3.client directly)

https://github.com/boto/boto3/pull/2746

https://github.com/rwillmer/boto3

Upvotes: 2

Andi-OTC
Andi-OTC

Reputation: 91

using boto3 resource:

import boto3

# use third party object storage
s3 = boto3.resource('s3', endpoint_url='https://URL:443',
  aws_access_key_id = 'AccessKey',
  aws_secret_access_key = 'SecertKey')

# Print out bucket names
for bucket in s3.buckets.all():
 print(bucket.name)

Upvotes: 8

petertc
petertc

Reputation: 3911

Another way:

import boto3

s3client = boto3.client('s3', endpoint_url='http://X.X.x.X:8080/',
        aws_access_key_id = 'XXXXXXX',
        aws_secret_access_key = 'XXXXXXXX')
bucket_name = 'aaaaa'
s3client.create_bucket(Bucket=bucket_name)

Upvotes: 9

Chillar Anand
Chillar Anand

Reputation: 29514

You cannot set host in config file, however you can override it from your code with boto3.

import boto3

session = boto3.session.Session()

s3_client = session.client(
    service_name='s3',
    aws_access_key_id='aaa',
    aws_secret_access_key='bbb',
    endpoint_url='http://localhost',
)

Then you can interact as usual.

print(s3_client.list_buckets())

Upvotes: 90

Jordon Phillips
Jordon Phillips

Reputation: 16003

boto3 only reads the signature version for s3 from that config file. You may want to open a feature request, but for now here is how you can address a custom endpoint:

import boto3
from botocore.utils import fix_s3_host
resource = boto3.resource(service_name='s3', endpoint_url='http://localhost')
resource.meta.client.meta.events.unregister('before-sign.s3', fix_s3_host)

That bit about the meta is important because boto3 automatically changes the endpoint to your_bucket_name.s3.amazonaws.com when it sees fit 1. If you'll be working with both your own host and s3, you may wish to override the functionality rather than removing it altogether.

Upvotes: 15

Related Questions