Trying to solve SAML interop issue between .NET and org.apache.santuario.xmlsec (specifically, Ping Federate)

Question

The issue is fairly well documented, however a solution from either Microsoft, or the community does not seem forthcoming.

When an identity provider platform utilising org.apache.santuario.xmlsec produces an xml signature, default character encoding causes what some service providers consider an invalid signature.

Issue discussed in quite a bit of depth, here, against WildFly (https://issues.redhat.com/browse/WFLY-9892) and here against Ping Federate (https://support.pingidentity.com/s/article/Newline-Characters-in-SAML-after-upgrade-to-PF-11).

I'm specifically seeing the issue validating signatures on a Windows .NET framework platform using SignedXml and Ping Federate.

Since the issue at first appears to be the possible handling of the canonicalization of the XML by the .NET framework, I started to pull out relevant parts of the guts to get to grips with what was going on under the hood.

The basic verification code looks like this:

    XmlDocument xmlDoc = new XmlDocument();
    xmlDoc.PreserveWhitespace = true;
    xmlDoc.Load(pathToXml);

    SignedXml signedXml = new SignedXml(xmlDoc);
    XmlNodeList nodeList = xmlDoc.GetElementsByTagName("ds:Signature");
    signedXml.LoadXml((XmlElement)nodeList[0]);
    var result = signedXml.CheckSignature();

This yields false for an XML document that I have from a Identity Provider producing additional 
 escape entities. But it is true for another example I have.

Tracing the CheckSignature() method, I can see that it's failing on Verifying SignedInfo

System.Security.Cryptography.Xml.SignedXml Verbose: 13 : [SignedXml#019251cf, VerifyReference] Reference Reference#03c72597 hashed with "http://www.w3.org/2001/04/xmlenc#sha256" (SHA256Managed) has hash value 5abb1948af860...216df2740d84106f0f00f, expected hash value 64caf89151e44bf156a6b15...b0adedbb045040e8e71.
System.Security.Cryptography.Xml.SignedXml Information: 12 : [SignedXml#019251cf, VerificationFailure] Verification failed checking references.

To try and rule out a bug in .NET, I started verifying the Signature's digest by parsing the XML (without the enveloped ds:Signature element) using an external C14n library.

Then calculating the SHA256 digest I can see that my code produces a match for the <ds:DigestValue>, so I know that the canonicalization library is perfect otherwise the calculated digest would be different.

I store this as a byte[] digestBytes for later on.

    <ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/>
    <ds:DigestValue>Zsr4fsHkh/FWprkUexhr6dRq8sAKhOhg3tjwRQQkjnE=

Verifying the signature should be an easy last step, right?

In the XML, the failing Signature looks like this, note the 
 escape chars, which are not valid for Base64, but are fine for the XML standard:

<ds:SignatureValue>
    Xxfh0hq34DGIQibeBrNcYuU/XD0aXGJmf3FEPb2pdOimjeWAo6BwLCo7LoS7QbPoqcH9Pl6JzJdI&#13;
    jgQei5AjrKV2mRbFzql51g5ALGkEK5Bxo81RIhQg+aPP3gLuJiZ6sNAdVKCxz3dmRtH81CXemMhM&#13;
    (snip)
    uxj5r1kHaBcLs83FvOcogleQhs5XLWI696Xf26FHdR0PVyr42BBxkRrY17c21asLcqzJapmn6Pzl&#13;
    ukC5udKmDmqS+Q0b8azdMo9MwjjVq257CR4Gx1uFaSaMJsDvh90hIviQm/5jZK6pD9wReldSPyo=
</ds:SignatureValue>

I pull the raw value of the signature out using an XmlDocument, which un-escapes the 
 escape chars allowing the Convert.FromBase64String() to correctly parse the string.

    var node = xmlDoc.SelectSingleNode("//ds:SignatureValue", nsManager);
    signatureValueBytes = Convert.FromBase64String(node.InnerText);

I then load the certificate as an X509Certificate2 in order to call VerifyHash on the signature, but it fails.

    var cyptoServiceProvider = (RSACryptoServiceProvider)verifyingCert.PublicKey.Key;
    cyptoServiceProvider.VerifyHash(digestBytes, "SHA256", signatureValueBytes)

At this point, I've ran out of ideas how to manually verify the signature. Going back to the signedXml.CheckSignature() call, at the deepest point, the .NET code calls into the unmanaged verification, so at this point it's difficult to know if something's going wrong in the parsing before this, or if it's further down the stack.

[DllImport("QCall", CharSet = CharSet.Unicode)]
[SecurityCritical]
[SuppressUnmanagedCodeSecurity]
private static extern bool VerifySign(SafeKeyHandle hKey, int calgKey, int calgHash, byte[] hash, int cbHash, byte[] signature, int cbSignature);