CODEWITHSUNDEEP
pythonmacosproxypython-requests
Amit Even
Amit Even

Reputation: 31

Python Requests or Urllib2 proxy setting ignored on OSX El Capitan 10.11.6

I consider my self an advanced developer when it comes to Python and Requests/Urllib2 libraries.

I have recently switched to Mac running both OSX and Win10 using Parallels. While running either one of the above mentioned libraries with it's proxy selection method, the OSX ignores the custom http request proxy settings and continues to use my public IP, this would not happen if i use Win10.

Method Attempts:

def simple_request(self):   
    base_url = "www.whatever.com"
    proxy = "http://192.168.2.1:8080"
    # i am using jsonip.com as a host to get my request IP   
    r = requests.get(r'http://jsonip.com', proxies={"http": proxy})   
    ip = r.json()['ip'] # will always be my public ip

I have been using proxies that were tested valid in two ways:

  1. Using Win10 python scripts applying the same method.

  2. Applying the same proxy settings to the OSX Networks Preferences Manager.

Moreover, i will never get the following error, even when using invalid proxy:

requests.exceptions.ConnectionError: HTTPConnectionPool  Max retries exceeded with url

I do get this error when i apply an invalid proxy settings to the OSX Networks Preferences Manager.

I have tried using proxies under osx with both Libraries mentioned and i get the same result.

Other solutions tried:

  1. Using Requests.Session manger to construct a request with proxy before sending it.

  2. Using the following https://stackoverflow.com/a/39934154/4255373 - it
    will change my settings but i'll have to restart my code to use it, and changing the connection setting to the whole OS services is not a valid solution for me.

Any suggestions?

Thanks.

Update: Using the logging module i get the following:

INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 199.116.113.206
send: 'GET http://jsonip.com/ HTTP/1.1\r\nHost: jsonip.com\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: python-requests/2.11.1\r\n\r\n'
reply: 'HTTP/1.1 301 Moved Permanently\r\n'
header: Date: Thu, 12 Jan 2017 02:13:25 GMT
header: Server: nginx/1.10.0 (Ubuntu)
header: Content-Type: text/html
header: Content-Length: 194
header: Location: https://jsonip.com/
header: Keep-Alive: timeout=5, max=100
header: Connection: Keep-Alive
DEBUG:requests.packages.urllib3.connectionpool:"GET http://jsonip.com/ HTTP/1.1" 301 194
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): jsonip.com
send: 'GET / HTTP/1.1\r\nHost: jsonip.com\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: python-requests/2.11.1\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx/1.10.0 (Ubuntu)
header: Date: Thu, 12 Jan 2017 02:13:25 GMT
header: Content-Type: application/json; charset=utf-8
header: Transfer-Encoding: chunked
header: Connection: keep-alive
header: Access-Control-Allow-Origin: *
header: Access-Control-Allow-Methods: GET
header: Strict-Transport-Security: max-age=31536000;
DEBUG:requests.packages.urllib3.connectionpool:"GET / HTTP/1.1" 200 None

Upvotes: 1

Views: 431

Answers (1)

Amit Even
Amit Even

Reputation: 31

Ok a rookie mistake.

i have followed l'L'l advice - simple but important - and after logging all urllib3 connections to the console i realized that I was trying to connect to an HTTPS server using an HTTP proxy and so it failed.

i still don't understand why the proxy is automatically ignored without an exception or warning printed to the console and why does the connection manager continue to send the request without a proxy.

Upvotes: 1

Related Questions