Flask Proxy Server Error in HTTPS - Works in HTTP but not HTTPS

Question

We have setup a basic proxy server on flask. It works fine in HTTP however when the request is made in HTTPS it throws up some issues that are proving difficult to debug.

It seems that the request is not correctly parsed by flask when in HTTPS.

A more detailed description is below

Server Code

from flask import Flask, request
import requests
from requests import exceptions


application = Flask(__name__)

@application.route("/",methods=["GET","POST","DELETE","PATCH","PUT"])
def route():
    try:
        response = requests.request(method=request.method, url=request.url, headers=request.headers,timeout=60)
        print(response.status_code, response.text)
        return (response.content, response.status_code, response.headers.items())
    except exceptions.ProxyError:
        return (b"Error", 408, request.headers.items())

if __name__ == "__main__":
    application.run(host="0.0.0.0", port=39100, debug=True, use_reloader=False,ssl_context=("certs/cert.pem","certs/key.pem"))

Request Sample Code

import requests

proxies = {"http":"https://127.0.0.1:39100", "https":"https://127.0.0.1:39100"}

# does not work if the request is http://ipv4.icanhazip.com/
# works if the request is http://ipv4.icanhazip.com/
testing = requests.get(url="https://ipv4.icanhazip.com/",proxies=proxies,verify=False) 
print(testing)
print(testing.content)

Error

/Users/a/PycharmProjects/invisiProxy/venv/bin/python /Users/a/PycharmProjects/invisiProxy/sudoRequest.py
Traceback (most recent call last):
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 700, in urlopen
    self._prepare_proxy(conn)
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 994, in _prepare_proxy
    conn.connect()
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/urllib3/connection.py", line 369, in connect
    self._tunnel()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/http/client.py", line 924, in _tunnel
    raise OSError(f"Tunnel connection failed: {code} {message.strip()}")
OSError: Tunnel connection failed: 404 NOT FOUND

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='ipv4.icanhazip.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 404 NOT FOUND')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/a/PycharmProjects/invisiProxy/sudoRequest.py", line 6, in <module>
    testing = requests.get(url="https://ipv4.icanhazip.com/",proxies=proxies,verify=False)
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/Users/a/PycharmProjects/invisiProxy/venv/lib/python3.10/site-packages/requests/adapters.py", line 559, in send
    raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='ipv4.icanhazip.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 404 NOT FOUND')))

Process finished with exit code 1

Answer 1

Using proxies on requests for HTTPS causes client to use HTTP tunneling via CONNECT method. That is necessary for client to have direct TCP channel to the target in order to establish end-to-end encryption without any MITM proxy (see urllib3 description for such case -- https://urllib3.readthedocs.io/en/stable/advanced-usage.html#http-and-https-proxies).

The CONNECT method is not supported by your flask server hence it returns 404 on the request. The behavior can be observed on the MVP server provided:

# client request http://ipv4.icanhazip.com/
127.0.0.1 - - [24/Jun/2022 12:08:12] "GET http://ipv4.icanhazip.com/ HTTP/1.1" 200 -

# client request https://ipv4.icanhazip.com/
127.0.0.1 - - [24/Jun/2022 12:03:48] "CONNECT ipv4.icanhazip.com:443 HTTP/1.0" 404

Technically, expected behavior is possible to perform ...

# openssl s_client --connect 127.0.0.1:39100
CONNECTED(00000003)
---
Certificate chain
 0 s:C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = xxxx
   i:C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = xxxx
---
read R BLOCK
GET https://ipv4.icanhazip.com/ HTTP/1.1
Host: ipv4.icanhazip.com

HTTP/1.1 200 OK
Server: Werkzeug/2.1.2 Python/3.7.3
...

IP_REDACTED
read:errno=0

... and such case is documented and "tweakable" on urllib3 level (used by requests for actual connections) with use_forwarding_for_https of https://urllib3.readthedocs.io/en/stable/reference/urllib3.poolmanager.html#urllib3.ProxyManager

BUT

• such usage is not generally recommended, because it violates the design of TLS itself (end-to-end encryption between client and resource).

• in current requests implementation there seem to be NO way to pass required flag from the developer via standard api. get_connection function would not allow to pass any additional proxy_kwargs to urllib3.ProxyManager

https://github.com/psf/requests/blob/da9996fe4dc63356e9467d0a5e10df3d89a8528e/requests/adapters.py#L352

    def get_connection(self, url, proxies=None):
        ...
        if proxies:
            ...
            proxy_manager = self.proxy_manager_for(proxy)
            conn = proxy_manager.connection_from_url(url)


https://github.com/psf/requests/blob/da9996fe4dc63356e9467d0a5e10df3d89a8528e/requests/adapters.py#L201
    def proxy_manager_for(self, proxy, **proxy_kwargs):
        ...
        manager = self.proxy_manager[proxy] = proxy_from_url(
                ...
                **proxy_kwargs,
            )

I would recommend to use full-flegged http proxy instead of flask server.