Erratic openssl/rsa behaviour: RSA_EAY_PRIVATE_DECRYPT:padding check failed

Question

I have written a program that runs permanently using <openssl/rsa> C library.
It basically decrypts a password given in argument. The problem is that sometimes it works flawlessly, and some other times it fails (with the same pubkey/privkey/password, returning this error:

message: error:04065072:rsa routines:RSA_EAY_PRIVATE_DECRYPT:padding check failed

Has anyone ever experienced that? Why this kind of error is returned, generally?

---

Some more details

It retrieves the private key at the initialisation of the program with the following:

#define PRIVFILE  "<correct-path>/privkey.pem"
EVP_PKEY *privKey;
int size_key;
FILE *fp = fopen(PRIVFILE, "r");
if (!fp) 
{
    <logs>
    return -1;
}
PEM_read_PrivateKey(fp, &privKey, 0, NULL);
fclose (fp);
 if (privKey == NULL)
{
    ERR_print_errors_fp (stderr);
    return -1;
}
size_key = EVP_PKEY_size(privKey);

Later, during a listening loop, a method call the private decryption algorithm

int len_enc = size_key;
unsigned char* enc_pw;
unsigned char* dec_pw;
int len_dec = 8;
char* err = malloc(130);
enc_pw = malloc(len_enc);
dec_pw = malloc(len_dec);
memset(enc_pw, 0, len_enc);
memset(dec_pw, 0, len_dec);
memcpy(enc_pw, value, len_enc); //value being the raw ciphered data to decrypt
ERR_load_crypto_strings();
if (RSA_private_decrypt(len_enc, enc_pw, dec_pw, privKey->pkey.rsa,RSA_PKCS1_OAEP_PADDING) == -1)
{
ERR_error_string(ERR_get_error(), err);
radlog(L_ERR, "message: %s", err);
}
free(enc_pw);
free(dec_pw);
free(err);

• I have done encryption on the data with perl using Crypt::OpenSSL::RSA:

  my $rsa_pub = Crypt::OpenSSL::RSA->new_public_key( $key_string);
  my $ciphertext = $rsa_pub->encrypt( $plaintext);

• There is some base64 encoding/decoding that i didn't mention to make it a little bit shorter. The problem does not come from that.

• private key and public key are generated with openssl genrsa: openssl genrsa -out privkey.pem 1024 and openssl rsa -in privkey.pem -pubout > pubkey.pub

It seems to work for some time, but occasionally (during a peak of request, if that matters) i get these errors for ciphered data that seemed valid before:

message: error:04065072:rsa routines:RSA_EAY_PRIVATE_DECRYPT:padding check failed

Answer 1

Is it a multi threaded application?

I was getting the same problem yesterday and, in my case, it was related to more than one thread using the key (one for decript and many others for encript). The problem was solved protecting the key with a mutex semaphore.

The service is up and stable since yesterday.