CODEWITHSUNDEEP
windowsopensslwindows-cetls1.2
sathish reddy s
sathish reddy s

Reputation: 31

How can I use Crypto APIs to sigh hash for TLS 1.2

I am working to upgrade TLS version form 1.0 to 1.2 for existing windows/Win CE application. Right now my application is using Openssl 0.9.8k(supports TLS 1.0). To upgrade TLS version I am using Openssl-1.0.2p(supports TLS 1.2). I was signing data using MD5-SHA1 similar to steps followed in MD5-SHA1 hash using CryptoAPI (my requirement was same as this)

  1. Created hash using Certificate key handle provider and use CALG_SSL3_SHAMD5 algorithm.
  2. Using CryptSetHashParam, I set the initial hash value
  3. Signed the hash using CryptSignHash.
  4. After that, reverse the bytes order (as OpenSSL signature is Big endian mentioned.

But TLS 1.2 has to sign hash using RSA-SHA256. How can I achieve using Crypto APIs?

I have tried to Created hash using Certificate key handle provider and use CALG_SHA_256 algorithm instead of CALG_SSL3_SHAMD5. But Certificate verification failed at server with error as "Bad Signature".

I have been surfing MSDN pages and I assuming that CNG APIs can do my Job for Windows 7 OS. If it is true, How can I achieve? Also I am not seeing CNG equivalent APIs in Windows CE, How can I achieve in Windows CE?

Thanks.

Upvotes: 1

Views: 155

Answers (1)

totti9848
totti9848

Reputation: 1

static RSA_METHOD *rsa_meth = NULL;
static int cng_create_rsa_method(void)
{
    if (rsa_meth == NULL) {
        if ((rsa_meth = RSA_meth_dup(RSA_get_default_method())) == NULL
            || !RSA_meth_set1_name(rsa_meth, "My RSA Provider")
            || !RSA_meth_set_flags(rsa_meth, 0)
            || !RSA_meth_set_priv_enc(rsa_meth, rsa_priv_enc)
            || !RSA_meth_set_sign(rsa_meth, rsa_sign)
            || !RSA_meth_set_init(rsa_meth, rsa_init)) {
            CNGerr(CNG_F_CNG_RSA_METHOD, ERR_R_MALLOC_FAILURE);
            RSA_meth_free(rsa_meth);
            rsa_meth = NULL;
        }
    }
    return rsa_meth != NULL;
}

Then the implementation method of rsa_sign will be:

int rsa_sign(int dtype, const unsigned char *m, unsigned int m_len,
                  unsigned char *sigret, unsigned int *siglen, const RSA *rsa) {
   HCRYPTHASH hHash = XXXX; //get the provide from your rsa;
   int ret = -1;
   std::vector<BYTE> output;
   switch (dtype) {
    case NID_sha256:
        alg = CALG_SHA_256;
        break;

    case NID_sha384:
        alg = CALG_SHA_384;
        break;

    case NID_sha512:
        alg = CALG_SHA_512;
        break;

    case NID_sha1:
        alg = CALG_SHA1;
        break;

    case NID_md5:
        alg = CALG_MD5;
        break;

    case NID_md5_sha1:
        alg = CALG_SSL3_SHAMD5;
        break;
   }
   if (CryptCreateHash(hProv, alg , 0, 0, &hHash)) {
      DWORD sigLen = 0;
      CryptSetHashParam(hHash, HP_HASHVAL, m, 0);
      CryptSignHash(hHash, AT_KEYEXCHANGE, NULL, 0, NULL, &sigLen);
      output.resize(sigLen);
      success = CryptSignHash(hHash, AT_KEYEXCHANGE, NULL, CRYPT_NOHASHOID, &output[0], &sigLen);
      CryptDestroyHash(hHash);
   }

   if (!output.empty()) {
      /*
       * Windows Crypto API result is little-endian but OpenSSL is big-endian so we must reverse it.
       */
      *siglen = output.size();
      unsigned char *sigPtr = sigret;
      for (std::vector<BYTE>::reverse_iterator iter = output.rbegin();
           iter != output.rend(); iter++) {
         *sigPtr = *iter;
         sigPtr++;
      }
      ret = 1;
   }
   return ret;
}

Upvotes: 0

Related Questions