CODEWITHSUNDEEP
python-3.xamazon-web-servicesproxybotocorepython-s3fs
MarlosB
MarlosB

Reputation: 475

Python AWS S3FS API: Manually set proxy server

I can't set proxy server for S3FS Python API. As S3FS's Config class is imported from botocore there is no S3FS documentation about it. So, I have read this question and also botocore documentation.

However, I couldn't manage to get botocore Config class working with S3FS.

I have tried:

from s3fs.core import Config
proxy = Config(proxies={'https':<my_proxy>})

fs = s3fs.S3FileSystem(key=MY_S3_KEY, secret=MY_S3_SECRET, config_kwargs=proxy)
fs.put(FILE_NAME, PATH)

I have also tried:

from s3fs.core import Config

fs = s3fs.S3FileSystem(key=MY_S3_KEY, secret=MY_S3_SECRET, config_kwargs=Config(proxies={'https':<my_proxy>}))
fs.put(FILE_NAME, PATH)

I have also tried to set same proxy for http. The proxy works fine from this computer, I have sucessfully tested several times from command line.

I always get Connection Timeout error message:

ConnectTimeoutError: Connect timeout on endpoint URL:....

Any suggestion is appreciated. Thanks

Upvotes: 1

Views: 1457

Answers (1)

sandlb
sandlb

Reputation: 198

Here are two ways to do it.

1 - Set an environment variable with the proxy:

os.environ['https_proxy'] = 'http://my_proxy_url'
s3 = s3fs.S3FileSystem(anon=False)

2 - pass in the proxy in the constructor:

s3 = s3fs.S3FileSystem(anon=False, config_kwargs={'proxies': {'https': 'http://my_proxy_url'}})

Upvotes: 2

Related Questions