CODEWITHSUNDEEP
copenssl
carloabelli
carloabelli

Reputation: 4349

Segmentation Fault Generating RSA Key Using OpenSSL

I am trying to generate RSA keys using OpenSSL with the following function:

RSA *genRSA() {
  clear();
  mvprintw(0, 0, "Generating RSA key...\n");
  RAND_load_file("/dev/random", 4096);
  BIGNUM *e = BN_new();
  BN_set_word(e, RSA_F4);
  RSA *rsa;
  while (getch() != '\n'); // the program does reach this point
  if (!RSA_generate_key_ex(rsa, 4096, e, 0)) { // seg fault must occur on this line
    while (getch() != '\n'); // never gets here
    printw("ERROR: Failed to create RSA key\n");
    return NULL;
  }
  while (getch() != '\n'); // or here
  BN_free(e);
  if (!RSA_check_key(rsa)) {
    printw("ERROR: Key failed validation\n");
    return NULL;
  }
  printw("Key generation completed successfully\n");
  return rsa;
}

I'm not receiving any compiler warnings other than some deprecated on OS X ones (could that be causing an issue?). Why am I getting a seg fault?

Upvotes: 2

Views: 1636

Answers (1)

PaulMcKenzie
PaulMcKenzie

Reputation: 35454

Without knowing anything about the library you're using, this is incorrect:

RSA *rsa;
while (getch() != '\n'); // the program does reach this point
  if (!RSA_generate_key_ex(rsa, 4096, e, 0))

You are calling RSA_generate_key_ex with an uninitialized pointer rsa. There is no way that RSA_generate_key_ex function is going to be able to do anything with it except attempt to use it and as you see, crash.

So read the docs on that function as to what the first parameter is supposed to be. Maybe it should be this:

RSA rsa;
while (getch() != '\n'); // the program does reach this point
  if (!RSA_generate_key_ex(&rsa, 4096, e, 0))

If this is the case, then you need to change your return type to RSA and not RSA* (I'm assuming that RSA is a struct or typedef of a type that can be returned safely by value).

Upvotes: 2

Related Questions