Python / Zeep / SOAP proxy problem (I think)

Question

I am trying to get this to work:

Practitioner notes -> Python scripts for automation in NA -> Python client to invoke NA SOAP APIs

Here is my code (sanitized a bit):

#! /usr/bin/env python3
from requests import Session
from zeep import Client
from zeep.transports import Transport

session = Session()
session.verify = False

transport = Transport(session=session)

client = Client( 'https://SERVER_FQDN/soap?wsdl=api.wsdl.wsdl2py', transport=transport)

# I added this for the network proxy
client.transport.session.proxies = {
        'http': '10.0.0.1:80',
        'https': '10.0.0.1:80',
        }

# Then found I needed this because "localhost" is hard-coded in the WSDL 
client.service._binding_options['address'] = 'https://SERVER_FQDN/soap' 

login_params = { 
            'username':'user',
            'password':'PASSWORD',
        }
loginResult = client.service.login(parameters=login_params )

sesnhdr_type = client.get_element('ns0:list_deviceInputParms')

sesnhdr = sesnhdr_type(sessionid=loginResult.Text)

devices = client.service.list_device(_soapheaders=[sesnhdr], parameters=sesnhdr)

print('\n\n ----------------------------- \n')
for i in devices.ResultSet.Row:
    print(i.hostName + ' ---> '+i.primaryIPAddress)
    params = {
            "ip":i.primaryIPAddress,
            "sessionid": loginResult.Text
            }
    device = client.service.show_deviceinfo(parameters=params)
    print(device.Text)
    print('\n\n ----------------------------- \n')

And here is my output:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 667, in urlopen
    self._prepare_proxy(conn)
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 932, in _prepare_proxy
    conn.connect()
  File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 317, in connect
    self._tunnel()
  File "/usr/lib64/python3.6/http/client.py", line 929, in _tunnel
    message.strip()))
OSError: Tunnel connection failed: 503 Service Unavailable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 727, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='SERVER_FQDN', port=443): Max retries exceeded with url: /soap (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 503 Service Unavailable',)))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./na-1.py", line XX, in <module>
    loginResult = client.service.login(parameters=login_params )
  File "/usr/local/lib/python3.6/site-packages/zeep/proxy.py", line 51, in __call__
    kwargs,
  File "/usr/local/lib/python3.6/site-packages/zeep/wsdl/bindings/soap.py", line 127, in send
    response = client.transport.post_xml(options["address"], envelope, http_headers)
  File "/usr/local/lib/python3.6/site-packages/zeep/transports.py", line 107, in post_xml
    return self.post(address, message, headers)
  File "/usr/local/lib/python3.6/site-packages/zeep/transports.py", line 74, in post
    address, data=message, headers=headers, timeout=self.operation_timeout
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 578, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 510, in send
    raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='SERVER_FQDN', port=443): Max retries exceeded with url: /soap (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 503 Service Unavailable',)))

I get the same errors if I use "localhost" and run the script on the server-in-question. The system has proxy environment values set. There are proper forward and reverse DNS entries for the server. The name and IP for the server are also in /etc/hosts

Here is the problem: If I use an IP address instead of the server's FQDN, the code runs.

Vendor support says the problem is not in their application that provides the endpoint:

The 503 error means that the service is not available, there are 3 situations that invoke this behavior: 1. The server is under maintenance, 2. The server is overloaded, 3. In rare cases, the DNS configuration is faulty. If we see, this problem is not related to NA because the request is working fine with the IP.

Any ideas on this ?

Why does only the IP work and NOT the FQDN or localhost ?

Answer 1

Most of the documentation I see for using proxies with Zeep start with client = Client(url) but that doesn't work if url is behind a firewall and can't be accessed except through a proxy! My attempt to do it according to the documentation did nothing but time out (of course).

The key is in understanding that Zeep is built on requests and requests can use proxies for initiating the session. So you need to build a proxied Session, then pass that session into the Transport, and initialize the Zeep client with that transport. This worked for me:

session = requests.Session()
session.auth = requests.auth.HTTPBasicAuth(soap_username, soap_password)
session.proxies = {"https": f"socks5://{settings.STATIC_PROXY}"}
transport = zeep.transports.Transport(session=session, timeout=(5, 30))
client = zeep.Client(url, transport=transport)

Answer 2

My problem lay in the fact that the initialization of Client wants to go ahead and make the connection, but I need the proxy setting at the start. So I cobbled together two examples from the official docs to set the proxy at the time the connection is made.

from zeep import Client
from zeep.transports import Transport
from requests import Session

session = Session()
session.proxies = {
    'http': 'http://username:password@proxy.example.com:8080',
    'https': 'http://username:password@proxy.example.com:8080'
}
transport=Transport(session=session)
client = Client(URL,transport=transport)