CODEWITHSUNDEEP
encryptiondsaencodingpkcs#11
LuxOnion
LuxOnion

Reputation:

Verifying DSA Signature generated by PKCS#11 with OpenSSL

I want to sign a SHA-256 hash with DSA using PKCS#11 Java Wrapper for the PKCS#11 API of a hardware security module. For this purpose I selected the Mechanism CKM_DSA, load the corresponding DSA key from the token and have the data (read as byte-array) signed. The key I use for testing has 1024bit length.

Everything seems to work fine: The key is loaded, the Session.sign() yields a byte[] array of length 40. This corresponds to the PKCS#11 Spec, which says:

"For the purposes of this mechanism, a DSA signature is a 40-byte string, corresponding to the concatenation of the DSA values r and s, each represented most significant byte first."

Now I want to verify this signature using openSSL, i.e., using

openssl dgst -d -sha256 -verify ${PUBLIC_KEY} -signature signature.der <raw input file>

This works if I

a) created the signature using OpenSSL

b) created the signature using bouncycastle and encoding the result as ASN1 encoded DER sequence.

Now I want to do the same with the PKCS#11 signature. My question is: how to format this 40 byte array? I tried the following:

        //sign data
        byte[] signedData = this.pkcs11Session.sign(dataToSign);
        //convert result
        byte[] r = new byte[20];
        byte[] s = new byte[20];
        System.arraycopy(signedData, 0, r, 0, 20);
        System.arraycopy(signedData, 19, s, 0, 20);

        //encode result
        ASN1EncodableVector v = new ASN1EncodableVector();
        v.add(new ASN1Integer(r));
        v.add(new ASN1Integer(s));
        return new DERSequence(v).getEncoded(ASN1Encoding.DER);

The encoding part seems to be correct, because it works if I produce r and s directly with bouncycastle and another software key. Besides, openssl does accept the input format but the verification fails sometimes with an error, sometimes just with "Verification failure".

Thus, I assume the conversion of the PKCS#11 signature to r and s is wrong. Can someone help finding the mistake?

Upvotes: 2

Views: 853

Answers (1)

Maarten Bodewes
Maarten Bodewes

Reputation: 94078

You probably have to convert the r and s values to a BigInteger class before you do. The reason for this is that ASN.1 uses signed value encoding and DH results in unsigned value encoding. So you've got a pretty high chance of getting a negative value in your ASN.1, which will result in an error.

To perform the conversion, use new BigInteger(1, r) and new BigInteger(1, s) and put the result into the ASN1Integer instances. Here 1 indicates that the value needs to be converted to a positive value (i.e. the input is unsigned positive).

Upvotes: 3

Related Questions