CODEWITHSUNDEEP
c++openssl
siqian Wei
siqian Wei

Reputation: 25

segmen fault for RSA_size() and EVP_PKEY_size() after use EVP_PKEY_assign_RSA() or EVP_PKEY_set1_RSA()

why RSA_size() and EVP_PKEY_size() will crash after i call EVP_PKEY_assign_RSA() or EVP_PKEY_set1_RSA() in my code?

openssl version is 1.10.

If i do not call EVP_PKEY_assign_RSA() or EVP_PKEY_set1_RSA(),program run correctly .

Also, I think those two function are both using to set RSA_st from EVP_PKEY,Am i got some misunderstanding of those two functions?

void testfun(char **argc) {

    OpenSSL_add_all_algorithms();

    EVP_PKEY *prikey = nullptr, *pubkey = nullptr;
    BIO *prifile = nullptr, *pubfile = nullptr;
    RSA *pubrsa = nullptr, *prirsa = nullptr, *newra = nullptr;

    prifile = BIO_new_file(argc[1], "r");

    char passwd[] = "1111";
    prikey = EVP_PKEY_new();
    prikey = PEM_read_bio_PrivateKey(prifile, nullptr, 0, passwd);

    prirsa = RSA_new();

    /* all those code block combination will cause segmentation fault
     * 1-3
     * 1-4
     * 2-3
     * 2-4
     * and output will be correct if i only use code block 3
     * */

    //1
    //cout << EVP_PKEY_assign_RSA(prikey, prirsa) << endl;

    //2
    //cout << EVP_PKEY_set1_RSA(prikey, prirsa) << endl;

    //3
    //cout << EVP_PKEY_size(prikey) << endl;

    //4
    //cout << RSA_size(prirsa) << endl;
}

Upvotes: 2

Views: 1297

Answers (1)

Leśny Rumcajs
Leśny Rumcajs

Reputation: 2516

RSA_size crash

RSA_size takes an RSA pointer and returns the key's modulus size - the provided key must be initialized, according to https://www.openssl.org/docs/man1.0.2/man3/RSA_size.html rsa->n must not be NULL. Your RSA_new does not create the key - it only allocates the structure. So you can't call a function that extracts size from a non-existent key.

EVP_PKEY_size crash

EVP_PKEY_size reason for crash is basically the same - you assign incorrect key to the existing, correct key (given that you provided a correct one in BIO_new_file.

So, the code that you probably seek is:

  1. Generate the RSA key with password 1111, like so:

    openssl genrsa -des3 -out private.pem 2048

  2. Run the program with the following code:

    #include <iostream>
#include <openssl/evp.h>
#include <openssl/pem.h>
#include <cassert>

int main() {
    OpenSSL_add_all_algorithms();

    // UNUSED
    // *pubkey = nullptr;
    // *pubfile = nullptr;
    // RSA *pubrsa = nullptr, *prirsa = nullptr, *newra = nullptr;

    BIO * prifile = BIO_new_file("../private.pem", "r");
    assert(prifile);

    char passwd[] = "1111";
    //prikey = EVP_PKEY_new(); // why allocate if you assign it somewhere else? memory leak!
    EVP_PKEY * prikey = PEM_read_bio_PrivateKey(prifile, nullptr, 0, passwd);
    assert(prikey);

    //1
    //std::cout << EVP_PKEY_assign_RSA(prikey, prirsa) << std::endl;

    //2
    //std::cout << EVP_PKEY_set1_RSA(prikey, prirsa) << std::endl;

    //3 Should correctly output 256 (modulo size in bytes)
    std::cout << EVP_PKEY_size(prikey) << std::endl;

    //4
    //std::cout << RSA_size(prirsa) << std::endl;

    std::cout << "done\n";
    return 0;
}

You don't need EVP_PKEY_assign_RSA nor EVP_PKEY_set1_RSA to read your key. The function PEM_read_bio_PrivateKey does all of that for you!

Upvotes: 1

Related Questions