Xml document signature not valid after deserialization from byte array

Question

I have a problem where I sign xml document and validate signature after that and the validation passes, but when I serialize document to byte array and then deserialize it back to document, signature validation fails.

Here are methods used for validation and serialization/deserialization:

public class DocumentSigner {
    @Override
    public byte[] transformToByteArray(Document doc) throws Exception {
        TransformerFactory transformerFactory = TransformerFactory
                .newInstance();
        Transformer transformer = transformerFactory.newTransformer();      
        ByteArrayOutputStream os = new ByteArrayOutputStream();
        transformer.transform(new DOMSource(doc), new StreamResult(os));
        return os.toByteArray();
    }

    private Document byteArrayToXmlDoc(byte[] documentoXml) throws Exception {
        DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
        factory.setNamespaceAware(true);
        factory.setIgnoringElementContentWhitespace(true);
        DocumentBuilder builder = factory.newDocumentBuilder();
        return builder.parse(new ByteArrayInputStream(documentoXml), "UTF-8");
    }

    @Override
    public Boolean validate(byte[] byteArrayDoc, Integer certificatePropertiesId) throws Exception {
        Document doc = byteArrayToXmlDoc(byteArrayDoc);
        return validate(doc, certificatePropertiesId);
    }

    public Boolean validate(Document doc, Integer certificatePropertiesId) throws Exception {
        NodeList nl = doc.getElementsByTagNameNS(XMLSignature.XMLNS,
                "Signature");
        if (nl.getLength() == 0) {
            throw new Exception("No signature element.");
        }

        KeyStore ks = KeyStore.getInstance("JKS");
        CertificatePropertiesDTO certProp = databaseLogic.getCertificateProperties(certificatePropertiesId);
        if (certProp == null || certProp.getCertificatePassword().isEmpty() || certProp.getCertificate() == null){
            throw new RuntimeException("No certificate.");
        }

        ks.load(new ByteArrayInputStream(certProp.getCertificate()), certProp.getCertificatePassword().toCharArray());
        KeyStore.PrivateKeyEntry keyEntry = (KeyStore.PrivateKeyEntry) ks.getEntry(ks.aliases().nextElement(), new KeyStore.PasswordProtection(certProp.getCertificatePassword().toCharArray()));
        X509Certificate[] certs =  (X509Certificate[]) keyEntry.getCertificateChain();
        if (certs == null || certs.length == 0) {
            throw new RuntimeException("No certificate found.");
        }

        XMLSignatureFactory fac = XMLSignatureFactory.getInstance("DOM");
        DOMValidateContext valContext = new DOMValidateContext(keyEntry.getCertificate().getPublicKey(), nl.item(0));
        NodeList els = doc.getElementsByTagNameNS("*", "SignatureProperties");
        Element el = (Element) els.item(0);
        valContext.setIdAttributeNS(el, null, "Id");
        valContext.setDefaultNamespacePrefix("dsig");

        valContext.setProperty("javax.xml.crypto.dsig.cacheReference", Boolean.TRUE);
        try {
            XMLSignature signature2 = fac
                    .unmarshalXMLSignature(new DOMStructure(nl.item(0)));
            boolean coreValidity = signature2.validate(valContext);

            // Check core validation status.
            if (coreValidity == false) {
                log.info("Signature failed core validation");
                boolean sv = signature2.getSignatureValue()
                        .validate(valContext);
                log.info("signature validation status: " + sv);
                Iterator<?> i = signature2.getSignedInfo().getReferences()
                        .iterator();
                for (int j = 0; i.hasNext(); j++) {
                    Reference ref = (Reference) i.next();
                    boolean refValid = ref.validate(valContext);
                    log.info("ref[" + j + "] validity status: " + refValid);
                }
                return false;
            } else {
                log.info("Signature passed core validation");
                return true;
            }
        } catch (Exception ex) {
            log.info("EXCEPTION during validation: " + ex.getMessage());
            return false;
        }
    }

    public void signDocument(Document doc)
    {
        ....
    }



public void writeToDisk(String path, String rac)
    {
        BufferedWriter writer = null;
        try
        {
            writer = new BufferedWriter(new FileWriter(path));
            writer.write(rac);

        }
        catch ( IOException e)
        {
        }
        finally
        {
            try
            {
                if ( writer != null)
                writer.close( );
            }
            catch ( IOException e)
            {
                try {
                    throw e;
                } catch (IOException e1) {
                    // TODO Auto-generated catch block
                    e1.printStackTrace();
                }
            }
        }
    }

    @Override
    public String transformToString(Document doc,
            Boolean omitXmlDeclaration) throws Exception {
        TransformerFactory transformerFactory = TransformerFactory
                .newInstance();
        //transformerFactory.setAttribute("indent-number", 4);
        Transformer transformer = transformerFactory.newTransformer();      
        if (omitXmlDeclaration)
            transformer.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION,
                    "yes");
//      transformer.setOutputProperty(OutputKeys.INDENT, "yes");
//      transformer.setOutputProperty(OutputKeys.ENCODING, "UTF-8");
//              
        StringWriter sw = new StringWriter();
        transformer.transform(new DOMSource(doc), new StreamResult(sw));
        //String output = sw.getBuffer().toString().replaceAll("\n|\r", ""); 
        return sw.toString();
    }
}

Here is where it passes/fails:

public void SignAndValidate()
{
    ...
    Document doc = createDocument();
    documentSigner.signDocument(doc);

    validate(doc, 1); 

    // OUTPUT:
    // Signature passed core validation

    byte[] docArr = documentSigner.transformToByteArray(doc);

    validate(docArr, 1);

    // OUTPUT:
    // Signature failed core validation
    // signature validation status: false
    // ref[0] validity status: false
    // ref[1] validity status: true
}

If necessary I will post methods for creating/signing document but it's big.

Here is the signing method:

private void signDocument(Document document) throws Exception {
        //Remove ApacheXMLDSig because after every SOAP message signing it's set as default provdier, but doesn't know about here() function from XPATH2
        Security.removeProvider("ApacheXMLDSig");

        XMLSignatureFactory sigFactory =  XMLSignatureFactory.getInstance("DOM", "XMLDSig");
        String id = String.format("id%s", UUID.randomUUID().toString());

        KeyStore ks = KeyStore.getInstance("JKS");
        CertificatePropertiesDTO certProp = databaseLogic.getCertificateProperties(1);
        if (certProp == null || certProp.getCertificatePassword().isEmpty() || certProp.getCertificate() == null){
            throw new RuntimeException("No certificate.");
        }

        ks.load(new ByteArrayInputStream(certProp.getCertificate()), certProp.getCertificatePassword().toCharArray());
        KeyStore.PrivateKeyEntry keyEntry = (KeyStore.PrivateKeyEntry) ks.getEntry(ks.aliases().nextElement(), new KeyStore.PasswordProtection(certProp.getCertificatePassword().toCharArray()));
        X509Certificate[] certs =  (X509Certificate[]) keyEntry.getCertificateChain();
        if (certs == null || certs.length == 0) {
            throw new RuntimeException("No certificate found.");
        }


        Element propSig = XMLElement(document, "PROP_Sig", "");
        Attr propNS = XMLAtribut(document, "xmlns", "http://ns.adobe.com/pdf/2006");
        propSig.setAttributeNodeNS(propNS);
        propSig.setAttribute("type", "cabinet");
        DateFormat df = new SimpleDateFormat("yyyyMMddHHmmssZZ");
        Element m = XMLElement(document, "M", String.format("D:%s", df.format(new Date())));
        m.setAttribute("type", "text");
        Element name = XMLElement(document, "Name", cert.getSubjectX500Principal().getName());      
        name.setAttribute("type", "text");
        propSig.appendChild(m);
        propSig.appendChild(name);
        SignatureProperty sp = sigFactory.newSignatureProperty(Collections.singletonList(new DOMStructure(propSig)), "data_signature", null);
        SignatureProperties sps = sigFactory.newSignatureProperties(Collections.singletonList(sp), id);

        CanonicalizationMethod cm = sigFactory.newCanonicalizationMethod(CanonicalizationMethod.EXCLUSIVE_WITH_COMMENTS, (XMLStructure) null);
        SignatureMethod sm = sigFactory.newSignatureMethod(SignatureMethod.RSA_SHA1, null);
        DigestMethod dm1 = sigFactory.newDigestMethod(DigestMethod.SHA1, null);
        Transform tf1 = sigFactory.newTransform(CanonicalizationMethod.EXCLUSIVE_WITH_COMMENTS, (TransformParameterSpec) null);     
        Reference ref1 = sigFactory.newReference("#" + id, dm1, Collections.singletonList(tf1), "http://www.w3.org/2000/09/xmldsig#SignatureProperties", null); 

        DigestMethod dm2 = sigFactory.newDigestMethod(DigestMethod.SHA1, null);
        String here = "here()/ancestor::dsig:Signature[1]/../../../../../..//. | "
                + "here()/ancestor::dsig:Signature[1]/../../../../../..//@* | "
                + "here()/ancestor::dsig:Signature[1]/../../../../../..//namespace::*";
        HashMap<String, String> hm = new HashMap<String, String>();
        hm.put("dsig", "http://www.w3.org/2000/09/xmldsig#");
        XPathType xp = new XPathType(here, XPathType.Filter.INTERSECT, hm);
        TransformParameterSpec paramsXPath2= new XPathFilter2ParameterSpec(Collections.singletonList(xp));
        Transform tf2 = sigFactory.newTransform(Transform.XPATH2, paramsXPath2);
        Transform tf3 = sigFactory.newTransform(Transform.ENVELOPED, (TransformParameterSpec) null);
        Transform tf4 = sigFactory.newTransform(CanonicalizationMethod.EXCLUSIVE_WITH_COMMENTS, (TransformParameterSpec) null); //"http://www.w3.org/2001/10/xml-exc-c14n#WithComments"
        List<Transform> lt2 = new ArrayList<Transform>();
        lt2.add(tf2);
        lt2.add(tf3);
        lt2.add(tf4);
        Reference ref2 = sigFactory.newReference("", dm2, lt2, null, null);

        List<Reference> lr = new ArrayList<Reference>();
        lr.add(ref1);
        lr.add(ref2);
        SignedInfo si = sigFactory.newSignedInfo(cm, sm, lr);

        KeyInfoFactory kif = KeyInfoFactory.getInstance("DOM", "XMLDSig");

        ArrayList<Object> x509Content = new ArrayList<>();
        for (int i = 0; i < certs.length; i++) {
            x509Content.add(certs[i]);
        }

        X509Data xd = kif.newX509Data(x509Content);
        KeyInfo ki = kif.newKeyInfo(Collections.singletonList(xd));

        DOMSignContext dsc = new DOMSignContext(keyEntry.getPrivateKey(), document.getElementsByTagName("sac:SignatureInformation").item(0));       
        XMLSignature signature = 
                sigFactory.newXMLSignature(si, ki, Collections.singletonList( sigFactory.newXMLObject(Collections.singletonList(sps), null, null, null)), "data_signature", null);

        dsc.setProperty("javax.xml.crypto.dsig.cacheReference", Boolean.TRUE);
        dsc.setDefaultNamespacePrefix("dsig");
        try {
            signature.sign(dsc);
        }
        catch (Exception ex) {
            log.warn(ex.getMessage());
            throw new RuntimeException("Signing failed");
        }
    }

Here is part of sample XML document which is signed:

        <ext:UBLExtension>
            <ext:ExtensionContent>
                <sig:UBLDocumentSignatures>
                    <sac:SignatureInformation>
                        <dsig:Signature xmlns:dsig="http://www.w3.org/2000/09/xmldsig#" Id="data_signature">
                            <dsig:SignedInfo>
                                <dsig:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#WithComments"/>
                                <dsig:SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/>
                                <dsig:Reference Type="http://www.w3.org/2000/09/xmldsig#SignatureProperties" URI="#idfe5688f4-583f-4a98-b26c-9d651b2f8918">
                                    <dsig:Transforms>
                                        <dsig:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#WithComments"/>
                                    </dsig:Transforms>
                                    <dsig:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/>
                                    <dsig:DigestValue>iq802IBHl5kVdIMWA5Wlb5hYEoY=</dsig:DigestValue>
                                </dsig:Reference>
                                <dsig:Reference URI="">
                                    <dsig:Transforms>
                                        <dsig:Transform Algorithm="http://www.w3.org/2002/06/xmldsig-filter2">
                                            <dsig:XPath Filter="intersect" xmlns:dsig="http://www.w3.org/2002/06/xmldsig-filter2">here()/ancestor::dsig:Signature[1]/../../../../../..//. | here()/ancestor::dsig:Signature[1]/../../../../../..//@* | here()/ancestor::dsig:Signature[1]/../../../../../..//namespace::*</dsig:XPath>
                                        </dsig:Transform>
                                        <dsig:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/>
                                        <dsig:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#WithComments"/>
                                    </dsig:Transforms>
                                    <dsig:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/>
                                    <dsig:DigestValue>2jmj7l5rSw0yVb/vlWAYkK/YBwk=</dsig:DigestValue>
                                </dsig:Reference>
                            </dsig:SignedInfo>
                            <dsig:SignatureValue>d+DRc25SXnhxwXJs10A9Hnf1g0gG2bZqqnpTbZvrzp8X3EvtOVr3dBP6Ldc1RMTJYSF+guanlWKn
liaKlu7VbdB+SiQRuAMAZt+9Cnbn0CMlIzt22nRJNzjbeLBpCm7K63jCHGOXsWCW43DI/DYeZwq+
Q2j7WESgOtWLqUO0Jn8=</dsig:SignatureValue>
                            <dsig:KeyInfo>
                                <dsig:X509Data>
                                    <dsig:X509Certificate>...</dsig:X509Certificate>
                                    <dsig:X509Certificate>...</dsig:X509Certificate>
                                </dsig:X509Data>
                            </dsig:KeyInfo>
                            <dsig:Object>
                                <dsig:SignatureProperties Id="idfe5688f4-583f-4a98-b26c-9d651b2f8918">
                                    <dsig:SignatureProperty Target="data_signature">
                                        <PROP_Sig xmlns="http://ns.adobe.com/pdf/2006" type="cabinet">
                                            <M type="text">D:20151130163741+0100</M>
                                            <Name type="text">CN=<CN>,L=<City>,O=<Organization>,C=<Country></Name>
                                        </PROP_Sig>
                                    </dsig:SignatureProperty>
                                </dsig:SignatureProperties>
                            </dsig:Object>
                        </dsig:Signature>
                    </sac:SignatureInformation>
                </sig:UBLDocumentSignatures>
            </ext:ExtensionContent>
        </ext:UBLExtension>
    </ext:UBLExtensions>

I don't understand why validation says reference[0] fails (the one which referes to element with id), but reference to whole document passes?

Answer 1

In my case it was the difference in the header value which is why it was failing.

The original xml document has and when it was written header was changed to

Which is why signature verification was failing.

Hence, remove xml declaration while parsing the doc and while writing the doc.

Answer 2

i had exactly the same issue as you have. signature was valid but references not. the problem is (at least in my case), that serialization and deserialization can affect the content of the xml. in my case it helped to call document.normalizeDocument() before signing the document and now the signature validates even after serialization/deserialization.

Answer 3

• Be sure their array character longs for the case are they equal? before the validation you must see some differencies.
• Also, some Signture technics can use any inner sign prefixes like a GUID, control them.

Use Utf8 on both

Answer 4

@formatc I don´t have privileges to comment, but can you try and view the hexadecimal values in both files (sign and deserialized). I had the same problem, for some reason in my case when constructing back the xml some non-visual characters are inserted in front of the document. You won´t see them unless you use HexView or some tool like that.

I was able to remove those characters programmatically and all went well.