Soap "Digital signature verification failure"

Question

I am trying to use the library zeep to access one of the service clients of a soap api for ERCOT (the texas energy grid - the api requires a certficiate). Here is my python code:

import contextlib
import os
import tempfile
from zeep import Client, Settings
from zeep.transports import Transport
from requests import Session
from requests_pkcs12 import Pkcs12Adapter
from zeep.wsse.signature import Signature
import random
import OpenSSL.crypto
import logging.config

# USE THE MOST VERBOSE LOGGING LEVEL
logging.config.dictConfig({
    'version': 1,
    'formatters': {
        'verbose': {
            'format': '%(name)s: %(message)s'
        }
    },
    'handlers': {
        'console': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
            'formatter': 'verbose',
        },
    },
    'loggers': {
        'zeep.transports': {
            'level': 'DEBUG',
            'propagate': True,
            'handlers': ['console'],
        },
    }
})

# Source: https://gist.github.com/erikbern/756b1d8df2d1487497d29b90e81f8068
@contextlib.contextmanager
def pfx_to_pem(pfx_path, pfx_password):
    ''' Decrypts the .pfx file to be used with requests. '''
    with tempfile.NamedTemporaryFile(suffix='.pem') as t_pem:
        f_pem = open(t_pem.name, 'wb')
        pfx = open(pfx_path, 'rb').read()
        p12 = OpenSSL.crypto.load_pkcs12(pfx, pfx_password)
        f_pem.write(OpenSSL.crypto.dump_privatekey(OpenSSL.crypto.FILETYPE_PEM, p12.get_privatekey()))
        f_pem.write(OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_PEM, p12.get_certificate()))
        ca = p12.get_ca_certificates()
        if ca is not None:
            for cert in ca:
                f_pem.write(OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_PEM, cert))
        f_pem.close()
        yield t_pem.name


def generate_nonce(length=15):
    """Generate pseudorandom number."""
    return ''.join([str(random.randint(0, 9)) for i in range(length)])

# CERTIFICATES PATHS
api_pfx_key = os.path.join('path to pfx cert here')
api_certificate = os.path.join('path to .cer file here')

# SETUP
wsdl_file = os.path.join('path to .wsdl file')
api_base_url = "https://testmisapi.ercot.com/2007-08/Nodal/eEDS/EWS/"
session = Session()
session.mount(api_base_url,
              Pkcs12Adapter(pkcs12_filename=api_pfx_key, pkcs12_password='XXXXXX'))
session.verify = True
transport = Transport(session=session)
settings = Settings(forbid_entities=False)

# CREATE CLIENT
print("Creating client.")
with pfx_to_pem(pfx_path=api_pfx_key, pfx_password=b'XXXXXX') as pem_file:
    client = Client(wsdl_file, settings=settings, transport=transport,
                    wsse=Signature(pem_file, api_certificate))

    print("Making request.")
    request_data = {
            "Header": {
                "Verb": "get",
                "Noun": "SystemStatus",
                "ReplayDetection": {
                    "Nonce": generate_nonce(),
                    "Created": "07-29-2019"},
                "Revision": "1",
                "Source": api_base_url,
                "UserID": "my_username",
                "MessageID": "00000000000",
                "Comment": "00000000000",
            },
        }
    print(client.service.Alerts(**request_data))

When I run this code, I get the following error, which is hit during print(client.service.Alerts(**request_data)):

  the_above_code.py, line 123, in <module>
    print(client.service.Alerts(**request_data))
  File "/anaconda3/envs/automate-bids-conda/lib/python3.7/site-packages/zeep/proxy.py", line 45, in __call__
    kwargs,
  File "/anaconda3/envs/automate-bids-conda/lib/python3.7/site-packages/zeep/wsdl/bindings/soap.py", line 130, in send
    return self.process_reply(client, operation_obj, response)
  File "/anaconda3/envs/automate-bids-conda/lib/python3.7/site-packages/zeep/wsdl/bindings/soap.py", line 185, in process_reply
    client.wsse.verify(doc)
  File "/anaconda3/envs/automate-bids-conda/lib/python3.7/site-packages/zeep/wsse/signature.py", line 73, in verify
    _verify_envelope_with_key(envelope, key)
  File "/anaconda3/envs/automate-bids-conda/lib/python3.7/site-packages/zeep/wsse/signature.py", line 313, in _verify_envelope_with_key
    signature = security.find(QName(ns.DS, "Signature"))
AttributeError: 'NoneType' object has no attribute 'find'

which is referencing this line in the zeep library: https://github.com/mvantellingen/python-zeep/blob/master/src/zeep/wsse/signature.py#L313

I have the most verbose logging level in, and here is the output when I try to access one of the service clients:

Creating client.
zeep.transports: Loading remote data from: http://www.w3.org/2001/xml.xsd
Making request.
zeep.transports: HTTP Post to https://testmisapi.ercot.com/2007-08/Nodal/eEDS/EWS/:
<?xml version='1.0' encoding='utf-8'?>
<soap-env:Envelope xmlns:soap-env="http://schemas.xmlsoap.org/soap/envelope/"><soap-env:Header><wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"><Signature xmlns="http://www.w3.org/2000/09/xmldsig#">
<SignedInfo>
<CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
<SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/>
<Reference URI={{ redacted - reference uri }}>
<Transforms>
<Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
</Transforms>
<DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/>
<DigestValue> {{redacted - digest value}} </DigestValue>
</Reference>
</SignedInfo>
<SignatureValue> {{redacted - signature value}} </SignatureValue>
<KeyInfo>
<wsse:SecurityTokenReference><X509Data>
<X509IssuerSerial>
<X509IssuerName>CN=DigiCert Global CA G2,O=DigiCert Inc,C=US</X509IssuerName>
<X509SerialNumber> {{redacted - serial number}} </X509SerialNumber>
</X509IssuerSerial>
<X509Certificate> {{redacted - my cert}} </X509Certificate>
</X509Data>
</wsse:SecurityTokenReference></KeyInfo>
</Signature></wsse:Security></soap-env:Header><soap-env:Body xmlns:ns1="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" ns1:Id={{ redacted - id}}><ns0:RequestMessage xmlns:ns0="http://www.ercot.com/schema/2007-06/nodal/ews/message"><ns0:Header><ns0:Verb>get</ns0:Verb><ns0:Noun>SystemStatus</ns0:Noun><ns0:ReplayDetection><ns0:Nonce>{{redacted - nonce}}</ns0:Nonce><ns0:Created>07-29-2019</ns0:Created></ns0:ReplayDetection><ns0:Revision>1</ns0:Revision><ns0:Source>https://testmisapi.ercot.com/2007-08/Nodal/eEDS/EWS/</ns0:Source><ns0:UserID> {{ redacted - my userid}} </ns0:UserID><ns0:MessageID>20110719SJ1</ns0:MessageID><ns0:Comment>{{ redacted - comment }}</ns0:Comment></ns0:Header></ns0:RequestMessage></soap-env:Body></soap-env:Envelope>
zeep.transports: HTTP Response from https://testmisapi.ercot.com/2007-08/Nodal/eEDS/EWS/ (status: 500):
<soap:Envelope
    xmlns:soap='http://schemas.xmlsoap.org/soap/envelope/'>
 <soap:Header>
 </soap:Header>
 <soap:Body>
  <soap:Fault>
   <faultcode>soap:Client</faultcode>
   <faultstring>SECU3504: Digital signature verification failure.
Validity of ds:SignedInfo's signature: false
Validaty of Signature references:
    #id-13a9f4ac-3f1a-4c69-bd08-cbcbc4ae2c4c: true</faultstring>
  </soap:Fault>
 </soap:Body>
</soap:Envelope>

What is causing this?

Answer 1

Did you manage to make some progress on that? I also came across this issue with ERCOT SOAP API.

Make sure that your admin issues a certificate for MOTE (test environment) rather than prod misAPI. Then, make sure to pass the certificate to both session and to the digital signature:

pkcs12 = PKCS12Manager(p12file='hdbkwhdkhee$API_testmyusername.pfx', passphrase='************')
session = Session()
session.cert = (pkcs12.get_cert(), pkcs12.get_key())
transport = Transport(session=session)
# --------------------------------------------------------------------------------------------------------------------
dig_signature = Signature(key_file=pkcs12.get_key(),  # private key part of the certificate
                          certfile=pkcs12.get_cert(),  # public key of the certificate - the certificate itself
                          password='******************')  # password generated from ERCOT UI when generating cert

Answer 2

I used BinarySignature instead of Signature with a modification to zeep to allow a separate cert/key for the signature verification. I essentially implemented this pull request

Answer 3

According to this link, certificate might not have been properly added at server side, so this is this a matter of contacting the issuer of the certificate, as opposed to a code change.

Github Issue