CODEWITHSUNDEEP
javac++openssldigital-signature
enzo
enzo

Reputation: 82

what is the type of data expected in RSA_verify, can't verify a signature generated from Java

I create a signature on Java using :

String message = "my message";
byte[] data = message.getBytes();
byte[] result;
Signature sig = Signature.getInstance("SHA512withRSA");
sig.initSign(this.privateKey);
sig.update(data);
result = sig.sign();

Then I save result as a Hex String into a text file, and using openssl I try to verify:

string signature//read hex string in the text file 
string message = "my message";
RSA *rsaPkey = NULL;

FILE *pemFile;
fopen_s(&pemFile, publicKeyFile, "r");
rsaPkey = PEM_read_RSA_PUBKEY(pemFile, &rsaPkey, NULL, NULL);
fclose(pemFile);

if(rsaPkey == NULL)
{
    RSA_free(rsaPkey);
    return 0;
}

int type                = NID_sha512;
const char *m           = message.c_str();
unsigned int m_len      = message.length();
int size                = RSA_size(rsaPkey);
unsigned char *sigret   = (unsigned char*) signature.c_str();
unsigned int siglen     = signature.length();

unsigned char digest[SHA512_DIGEST_LENGTH];

SHA512_CTX ctx;
SHA512_Init(&ctx);
SHA512_Update(&ctx, m, m_len);
SHA512_Final(digest, &ctx);

int r = RSA_verify(type, digest, SHA512_DIGEST_LENGTH, sigret, siglen, rsaPkey);

r is always 0, which means the verification has failed. I think it's because it's expecting the message or the signature in a specific format than the hex I got from Java but i don't know what exaclty.

UPDATE

After using the digest as suggested by Pras, I get this message error when I use ERR_get_error : error:04091068:rsa routines:INT_RSA_VERIFY:bad signature

Upvotes: 0

Views: 1273

Answers (2)

Pras
Pras

Reputation: 4044

From RSA_verify() man page:

RSA_verify() verifies that the signature sigbuf of size siglen matches a given message digest m of size m_len. type denotes the message digest algorithm that was used to generate the signature. rsa is the signer's public key.

So the second and third arguments are supposed to be message digest and digest length not actual message and message length that was signed

Upvotes: 1

Jonathan Rosenne
Jonathan Rosenne

Reputation: 2217

It could be that byte[] data = message.getBytes(); may not give the same sequence of bytes as message.c_str(). I suggest you check it with a debugger.

Upvotes: 0

Related Questions