Using DER key from Java in mbedtls Diffie-Hellman

Question

I have a Java application that is producing keys for a Diffie-Hellman key exchange. These keys are generated, and the public key is exported as follows:

/*
 * Alice creates her own DH key pair with 2048-bit key size
 */
System.out.println("ALICE: Generate DH keypair ...");
KeyPairGenerator aliceKpairGen = KeyPairGenerator.getInstance("DH");
aliceKpairGen.initialize(2048);
KeyPair aliceKpair = aliceKpairGen.generateKeyPair();

// Alice creates and initializes her DH KeyAgreement object
System.out.println("ALICE: Initialization ...");
KeyAgreement aliceKeyAgree = KeyAgreement.getInstance("DH");
aliceKeyAgree.init(aliceKpair.getPrivate());

// Alice encodes her public key, and sends it over to Bob.
byte[] alicePubKeyEnc = aliceKpair.getPublic().getEncoded();

Contained in alicePubKeyEnc is the public key I am sending over to my C++ application. As I understand it, this data is an encoded DER key for my DHE parameters. On the other side I am attempting to use this key with mbedtls as follows:

// An example DER (?) key from Java.
unsigned char buf[] = {
    0x30, 0x82, 0x02, 0x28, 0x30, 0x82, 0x01, 0x1b, 0x06, 0x09, 0x2a, 0x86,
    0x48, 0x86, 0xf7, 0x0d, 0x01, 0x03, 0x01, 0x30, 0x82, 0x01, 0x0c, 0x02,
    0x82, 0x01, 0x01, 0x00, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
    0xc9, 0x0f, 0xda, 0xa2, 0x21, 0x68, 0xc2, 0x34, 0xc4, 0xc6, 0x62, 0x8b,
    0x80, 0xdc, 0x1c, 0xd1, 0x29, 0x02, 0x4e, 0x08, 0x8a, 0x67, 0xcc, 0x74,
    0x02, 0x0b, 0xbe, 0xa6, 0x3b, 0x13, 0x9b, 0x22, 0x51, 0x4a, 0x08, 0x79,
    0x8e, 0x34, 0x04, 0xdd, 0xef, 0x95, 0x19, 0xb3, 0xcd, 0x3a, 0x43, 0x1b,
    0x30, 0x2b, 0x0a, 0x6d, 0xf2, 0x5f, 0x14, 0x37, 0x4f, 0xe1, 0x35, 0x6d,
    0x6d, 0x51, 0xc2, 0x45, 0xe4, 0x85, 0xb5, 0x76, 0x62, 0x5e, 0x7e, 0xc6,
    0xf4, 0x4c, 0x42, 0xe9, 0xa6, 0x37, 0xed, 0x6b, 0x0b, 0xff, 0x5c, 0xb6,
    0xf4, 0x06, 0xb7, 0xed, 0xee, 0x38, 0x6b, 0xfb, 0x5a, 0x89, 0x9f, 0xa5,
    0xae, 0x9f, 0x24, 0x11, 0x7c, 0x4b, 0x1f, 0xe6, 0x49, 0x28, 0x66, 0x51,
    0xec, 0xe4, 0x5b, 0x3d, 0xc2, 0x00, 0x7c, 0xb8, 0xa1, 0x63, 0xbf, 0x05,
    0x98, 0xda, 0x48, 0x36, 0x1c, 0x55, 0xd3, 0x9a, 0x69, 0x16, 0x3f, 0xa8,
    0xfd, 0x24, 0xcf, 0x5f, 0x83, 0x65, 0x5d, 0x23, 0xdc, 0xa3, 0xad, 0x96,
    0x1c, 0x62, 0xf3, 0x56, 0x20, 0x85, 0x52, 0xbb, 0x9e, 0xd5, 0x29, 0x07,
    0x70, 0x96, 0x96, 0x6d, 0x67, 0x0c, 0x35, 0x4e, 0x4a, 0xbc, 0x98, 0x04,
    0xf1, 0x74, 0x6c, 0x08, 0xca, 0x18, 0x21, 0x7c, 0x32, 0x90, 0x5e, 0x46,
    0x2e, 0x36, 0xce, 0x3b, 0xe3, 0x9e, 0x77, 0x2c, 0x18, 0x0e, 0x86, 0x03,
    0x9b, 0x27, 0x83, 0xa2, 0xec, 0x07, 0xa2, 0x8f, 0xb5, 0xc5, 0x5d, 0xf0,
    0x6f, 0x4c, 0x52, 0xc9, 0xde, 0x2b, 0xcb, 0xf6, 0x95, 0x58, 0x17, 0x18,
    0x39, 0x95, 0x49, 0x7c, 0xea, 0x95, 0x6a, 0xe5, 0x15, 0xd2, 0x26, 0x18,
    0x98, 0xfa, 0x05, 0x10, 0x15, 0x72, 0x8e, 0x5a, 0x8a, 0xac, 0xaa, 0x68,
    0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x02, 0x01, 0x02, 0x02,
    0x02, 0x04, 0x00, 0x03, 0x82, 0x01, 0x05, 0x00, 0x02, 0x82, 0x01, 0x00,
    0x2f, 0x6e, 0xa3, 0xd8, 0x5c, 0xca, 0x06, 0x99, 0xbd, 0x35, 0x90, 0xb5,
    0xc3, 0x11, 0xa6, 0x48, 0x5b, 0x5a, 0xe9, 0x14, 0xac, 0x4a, 0xed, 0x2d,
    0x46, 0xb1, 0x6b, 0xc7, 0x5b, 0x88, 0xc6, 0xa7, 0x51, 0x07, 0xa0, 0x4d,
    0x5f, 0xc0, 0x32, 0x54, 0x9d, 0x63, 0x35, 0xa2, 0x3b, 0x6a, 0x9a, 0x0a,
    0xb9, 0x46, 0xff, 0x0b, 0x78, 0x5d, 0xa7, 0x17, 0x94, 0x58, 0x28, 0x28,
    0xf2, 0xa0, 0xea, 0x3a, 0xf0, 0xe8, 0x8f, 0xc2, 0xa1, 0x7c, 0xb1, 0x50,
    0x38, 0xb7, 0x01, 0xe1, 0x69, 0x42, 0x30, 0x1e, 0x06, 0x06, 0x06, 0x17,
    0x46, 0xc8, 0x1f, 0xb1, 0xb4, 0xd2, 0xff, 0xf1, 0x32, 0xdc, 0xc2, 0xfc,
    0x2c, 0x15, 0xe4, 0xfe, 0xae, 0xb2, 0x1f, 0x8b, 0x20, 0x29, 0x87, 0xbe,
    0x31, 0x8c, 0xf2, 0x01, 0x95, 0x51, 0x35, 0x76, 0x4c, 0x83, 0xe4, 0x06,
    0x46, 0x96, 0x62, 0x42, 0x2f, 0x23, 0xb4, 0xb7, 0xc7, 0x41, 0x4b, 0x4e,
    0xf5, 0xab, 0x20, 0xb0, 0x45, 0x27, 0x52, 0x64, 0x63, 0x18, 0x87, 0x72,
    0xa7, 0x41, 0x80, 0xbd, 0x15, 0x4d, 0xa8, 0x48, 0x69, 0x69, 0x8b, 0x64,
    0x38, 0x03, 0xa7, 0x72, 0xf7, 0xeb, 0x2b, 0xdd, 0x19, 0x2d, 0x63, 0x3a,
    0xa6, 0x1b, 0x6f, 0xcc, 0x81, 0x14, 0xde, 0x29, 0xd9, 0x55, 0x66, 0xd1,
    0x95, 0x8d, 0x2f, 0x15, 0x5e, 0x29, 0xad, 0xf8, 0x82, 0xf0, 0x68, 0xac,
    0x65, 0xf7, 0x54, 0x4f, 0x3e, 0x52, 0x64, 0xe8, 0x28, 0x52, 0x0f, 0x7c,
    0xbe, 0xc4, 0xf2, 0x20, 0x40, 0x97, 0xfa, 0x0a, 0x78, 0x5b, 0x1d, 0xf8,
    0xdb, 0x15, 0x02, 0xbe, 0xc3, 0xf0, 0xb0, 0x3a, 0xda, 0x6a, 0xe6, 0x5f,
    0x74, 0x48, 0x74, 0x0e, 0xe9, 0x1d, 0x02, 0xda, 0x25, 0x37, 0x4f, 0x41,
    0x11, 0x63, 0x32, 0x93, 0x44, 0xfe, 0x5b, 0x3a, 0x7e, 0x25, 0xcc, 0x9f,
    0xd5, 0x99, 0x41, 0x9f, 0x00
};

size_t size = 557; // 556 + NULL terminating byte per the docs.

mbedtls_dhm_context ctx;
mbedtls_dhm_init(&ctx);
int res = mbedtls_dhm_parse_dhm(&ctx, buf, size);

if (res != 0) {
    printf("FAIL\n", -res);
}

res results in a negative, non-zero value (0xffffcc1e). I've checked the error codes for x509 parsing in mbedtls, but can't find a matching error code.

Any idea why I can't load this certificate?

Update

I updated my C code to print out the associated error message for mbedtls:

0x0000023c4f9fb2b0 "DHM - The ASN.1 data is not formatted correctly : ASN1 - ASN1 tag was of an unexpected value"

So I took the hex values from my key and ran it through http://lapo.it/asn1js/ (<-- That links the the actual output from the key) which indicates some structure to my key and seems to parse the key just fine. Perhaps it's an issue with how I'm passing the key to mbedtls?

Answer 1

You are passing the wrong thing to mbed TLS's mbedtls_dhm_parse_dhm. That function expects the DER (or PEM) encoded DH parameters, not the key.

Using BouncyCastle's DHParameter you can get the DER encoded parameters like this:

DHParameterSpec dhParams = ((DHPublicKey) aliceKeyPair.getPublic()).getParams();
DHParameter dhP = new DHParameter(dhParams.getP(), dhParams.getG(), 0);
byte[] encodedParams = dhP.getEncoded(ASN1Encodable.DER);

This will give you the bytes that you can then pass into mbedtls_dhm_parse_dhm, either as raw bytes as you are doing now or PEM encoded as that function will accept that as well. Note that I'm passing 0 for L as mbed TLS discards it anyway. Also note that getEncoded throws so make sure to handle the exception.

Steps to complete a key exchange:

In Java:

KeyPairGenerator aliceKpairGen = KeyPairGenerator.getInstance("DH");
aliceKpairGen.initialize(2048);
KeyPair aliceKpair = aliceKpairGen.generateKeyPair();
KeyAgreement aliceKeyAgree = KeyAgreement.getInstance("DH");
aliceKeyAgree.init(aliceKpair.getPrivate());
DHParameterSpec dhParams = ((DHPublicKey) aliceKpair.getPublic()).getParams();
DHParameter dhP = new DHParameter(dhParams.getP(), dhParams.getG(), 0);
byte[] encodedParams = dhP.getEncoded(ASN1Encodable.DER);
byte[] javaPub = ((DHPublicKey)aliceKeyPair.getPublic()).getY().toByteArray();

Then in C++:

mbedtls_dhm_context ctx;
mbedtls_dhm_init(&ctx);
...
mbedtls_dhm_parse_dhm(&ctx, encodedParams, encodedParamsLen);
...
uint_8t mbedtlspub[128];
mbedtls_dhm_make_public(&ctx, 128, mbedtlspub, sizeof(mbedtlspub), mbedtls_ctr_drbg_random, &rnd_info);
...
mbedtls_dhm_read_public(&ctx, javaPub, javaPubLen);
...
uint_8t mbedtlssecret[128];
mbedtls_dhm_calc_secret(&ctx, mbedtlssecret, (size_t)secret_len, &olen, mbedtls_ctr_drbg_random, &rnd_info);

And back in Java (assuming you encoded mbedtlspub from the previous steps in Base64, but you can do it however you want):

byte[] yBinary = Base64.decodeBase64(mbedtlspub.getBytes()); 
BigInteger y = new BigInteger(yBinary); 
DHPublicKeySpec dhPublicKeySpec = new DHPublicKeySpec(y, dhParams.getP(), dhParams.getG()); 
KeyFactory keyFactory = KeyFactory.getInstance(ALGORITHM); 
DHPublicKey mbedtlsPubKey = (DHPublicKey) KeyFactory.generatePublic(dhPublicKeySpec);
aliceKeyAgree.doPhase(mbedtlsPubKey, true);
byte[] javaSecretKey = aliceKeyAgree.generateSecret();

That completes the key exchange and at this point mbedtlssecret and javaSecretKey should match. So the data passing hands is encodedParams and javaPub (from Java) and mbedtlspub (from C++). I omitted some detail to focus on the core steps. Also note that a lot of these functions either return an error code or throw, so please take care using them.