Openssl library equivalent of java KeyPairGenerator

Question

I'm trying to send a RSA public key to a Java client from a c++ server. Since the client is running Java I believe the key needs to be formatted in X509. I need to get the key as an encoded char array but can't find the right way.

I need to find the c++ openssl library equivalent of the following:

public byte[] getEncodedPubKey() {
    try {
        KeyPairGenerator keypairgenerator = KeyPairGenerator.getInstance("RSA");
        keypairgenerator.initialize(1024);
        return keypairgenerator.generateKeyPair().getPublic().getEncoded();
    } catch (Exception exception) {
        //Insert error handling
    }
}

This my best attempt in c++ at the solution so far:

EVP_PKEY *evpKey = EVP_PKEY_new();
RSA *rsa = RSA_generate_key(1024, RSA_F4, nullptr, nullptr);
EVP_PKEY_assign_RSA(evpKey, rsa);

X509 *x = X509_new();
X509_set_pubkey(x, evpKey);

auto *data = (unsigned char*)malloc(165);
X509_PUBKEY *pubkey = X509_get_X509_PUBKEY(x);
int size = i2d_X509_PUBKEY(pubkey, &data);

std::vector<unsigned char> encodedKey(data, data+size);

To provide more information, this is what java client does once receiving the encoded public key:

public PublicKey decodePubKey(byte[] encodedKeyIn) {
    try {
        KeyFactory keyfactory = KeyFactory.getInstance("RSA");
        EncodedKeySpec encodedkeyspec = new X509EncodedKeySpec(encodedKeyIn);
        return keyfactory.generatePublic(encodedkeyspec);
    } catch (Exception exception) {
        //Insert error handling
    }
}

The size of the encoded byte array generated by my c++ code is the same as the java example. Yet, the client still throws this error: error: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: IOException: DerInputStream.getLength(): lengthTag=77, too big.

Any ideas on what I am doing wrong? Thanks in advance

Answer 1

This code, edited from @Afshin's post, generates a RSA key pair and populates a std::vector<unsigned char> with the public key's PEM data base64 decoded. The data in the vector is equivalent to the data in the byte[] generated by the java function getEncodedPubKey() posted in the quesiton. The openssl function that solved my problem was PEM_read_bio() its documentation is here PEM_read_bio. This function decodes the key data in the PEM formatted BIO.

Here is the code:

#include <iostream>
#include <vector>
#include <memory>
#include <openssl/evp.h>
#include <openssl/rsa.h>
#include <openssl/bio.h>
#include <openssl/pem.h>

struct BioDeleter {
      void operator()(BIO* bio) { BIO_free(bio); }
};

struct KeyCtxDeleter {
      void operator()(EVP_PKEY_CTX* ctx) { EVP_PKEY_CTX_free(ctx); }
};

using BIO_ptr = std::unique_ptr<BIO, BioDeleter>;
using PKEY_CTX_ptr = std::unique_ptr<EVP_PKEY_CTX, KeyCtxDeleter>;

int main() {
    EVP_PKEY *pkey = nullptr;

    PKEY_CTX_ptr ctx(EVP_PKEY_CTX_new_id(EVP_PKEY_RSA, nullptr));

    EVP_PKEY_keygen_init(ctx.get());

    EVP_PKEY_CTX_set_rsa_keygen_bits(ctx.get(), 1024);

    EVP_PKEY_keygen(ctx.get(), &pkey);

    BIO_ptr bio(BIO_new(BIO_s_mem()));
    PEM_write_bio_PUBKEY(bio.get(), pkey);
    EVP_PKEY_free(pkey);

    long size = 0;
    char* dummy1 = NULL;
    char* dummy2 = NULL;
    unsigned char* data = NULL;
    PEM_read_bio(bio.get(), &dummy1, &dummy2, &data, &size);
    OPENSSL_free(dummy1);
    OPENSSL_free(dummy2);

    std::vector<unsigned char> decodedKey(data, data+size);
    OPENSSL_free(data);
}

Note: There is no error handling as this is just an example

Answer 2

This small code generates a RSA key-pair and stores public key into a string with PEM format. Check if it solves your problem:

#include <iostream>
#include <string>
#include <memory>
#include <openssl/evp.h>
#include <openssl/rsa.h>
#include <openssl/bio.h>
#include <openssl/pem.h>

struct BioDeleter {
      void operator()(BIO* bio) { BIO_free(bio); }
};

struct KeyCtxDeleter {
      void operator()(EVP_PKEY_CTX* ctx) { EVP_PKEY_CTX_free(ctx); }
};

using BIO_ptr = std::unique_ptr<BIO, BioDeleter>;
using PKEY_CTX_ptr = std::unique_ptr<EVP_PKEY_CTX, KeyCtxDeleter>;

int main() {
  EVP_PKEY *pkey = nullptr;

  PKEY_CTX_ptr ctx(EVP_PKEY_CTX_new_id(EVP_PKEY_RSA, nullptr));
  if (!ctx)
    return -1;

  if (EVP_PKEY_keygen_init(ctx.get()) <= 0)
    return -1;

  if (EVP_PKEY_CTX_set_rsa_keygen_bits(ctx.get(), 2048) <= 0)
    return -1;

  if (EVP_PKEY_keygen(ctx.get(), &pkey) <= 0)
    return -1;

  BIO_ptr bio(BIO_new(BIO_s_mem()));
  auto res = PEM_write_bio_PUBKEY(bio.get(), pkey);
  EVP_PKEY_free(pkey);

  if (res <= 0)
    return -1;

  BUF_MEM* mem = nullptr;
  BIO_get_mem_ptr(bio.get(), &mem);
  auto str = std::string{mem->data, mem->data + mem->length};
  std::cout << str << "\n";
}

In addition, you can read PEM is java side like this:

public static RSAPublicKey readPublicKey(File file) throws Exception {
    String key = new String(Files.readAllBytes(file.toPath()), Charset.defaultCharset());

    String publicKeyPEM = key
      .replace("-----BEGIN PUBLIC KEY-----", "")
      .replaceAll(System.lineSeparator(), "")
      .replace("-----END PUBLIC KEY-----", "");

    byte[] encoded = Base64.decodeBase64(publicKeyPEM);

    KeyFactory keyFactory = KeyFactory.getInstance("RSA");
    X509EncodedKeySpec keySpec = new X509EncodedKeySpec(encoded);
    return (RSAPublicKey) keyFactory.generatePublic(keySpec);
}

This code comes from here.