Cannot decrypt in C# an encrypted string done by openssl. Always get Org.BouncyCastle.Security.InvalidKeyException

Question

I trying to decrypt a string in C# that was encrypted by openssl via command line but it keep thrown the exception below:

Exception thrown: 'Org.BouncyCastle.Security.InvalidKeyException' in BouncyCastle.Crypto.dll
An unhandled exception of type 'Org.BouncyCastle.Security.InvalidKeyException' occurred in 
BouncyCastle.Crypto.dll
Not an RSA key

I am using RSA encryption and these are the commands (from git bash) that I use to generate the .pem file sand encrypt/decrypt a test string:

// Create private key
// openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -pkeyopt rsa_keygen_pubexp:65537 -out privkey.pem

// Create public key
// openssl pkey -pubout -in privkey.pem -out pubkey.pem

// "helloworld" as test text
// echo helloworld > clear_text.txt

// Encrypt using public key
// openssl rsautl -encrypt -inkey pubkey.pem -pubin -in clear_text.txt -out crypt.bin

// Decrypt using private key
// openssl rsautl -decrypt -inkey privkey.pem -in crypt.bin -out decrypt.txt

What need to do in C# is just the last line of the sequence above but in C#. Here is the C# code that I use to load the .pem files (generated by openssl) and decrypt the string.

    static private PemReader publicKeyPemFile;
    static private PemReader privateKeyPemFile;
    static private string filesRootPath = System.IO.Directory.GetCurrentDirectory() + "/../../../";

    static public void LoadKeys()
    {
        publicKeyPemFile = new PemReader(
            (StreamReader)File.OpenText(filesRootPath + "pubkey.pem")
        );

        privateKeyPemFile = new PemReader(
            (StreamReader)File.OpenText(filesRootPath + "privkey.pem")
        );
    }


    static public string RSADecrypt(byte[] cipherTextBytes)
    {
        RsaKeyParameters keys = (RsaKeyParameters)privateKeyPemFile.ReadObject();

        OaepEncoding eng = new OaepEncoding(new RsaEngine());
        eng.Init(false, keys);

        int length = cipherTextBytes.Length;
        int blockSize = eng.GetInputBlockSize();
        List<byte> plainTextBytes = new List<byte>();
        for (int chunkPosition = 0;
            chunkPosition < length;
            chunkPosition += blockSize)
        {
            int chunkSize = Math.Min(blockSize, length - chunkPosition);
            plainTextBytes.AddRange(eng.ProcessBlock(
                cipherTextBytes, chunkPosition, chunkSize
            ));
        }

        string outString = Encoding.UTF8.GetString(plainTextBytes.ToArray());
        return outString;
    }

but when trying to decrypt it fails as below with the exception.

What really bugs me is that if I encrypt and decrypt using C# only using the encryption below, it works. Here is a C# encryption code that uses the .pem files generated by openssl.

    static public string RSAEncrypt(string clearText)
    {
        byte[] plainTextBytes = Encoding.UTF8.GetBytes(clearText);

        RsaKeyParameters keys = (RsaKeyParameters)publicKeyPemFile.ReadObject();

        OaepEncoding eng = new OaepEncoding(new RsaEngine());
        eng.Init(true, keys);

        int length = plainTextBytes.Length;
        int blockSize = eng.GetInputBlockSize();
        List<byte> cipherTextBytes = new List<byte>();
        for (int chunkPosition = 0;
            chunkPosition < length;
            chunkPosition += blockSize)
        {
            int chunkSize = Math.Min(blockSize, length - chunkPosition);
            cipherTextBytes.AddRange(eng.ProcessBlock(
                plainTextBytes, chunkPosition, chunkSize
            ));
        }

        return result;Convert.ToBase64String(cipherTextBytes.ToArray());
    }

but if encrypt with openssl and try to decrypt with C# is always crashes.

The final test program

    static void Main(string[] args)
    {
        RSAEncryption.LoadKeys();
        RSAEncryption.TestRSAFromOpenSSL();
    }

    static public void TestRSAFromOpenSSL()
    {
        byte[] bytes = File.ReadAllBytes(filesRootPath + "crypt.bin");
        string outString = RSADecrypt(bytes);

        Debug.WriteLine("clear text = " + bytes.Length);
    }

Any clues?

thank you.

Answer 1

I tried replicating your problem with a simpler piece of code and I can't. But I got an error when trying to decrypt the text. It turns out that my openssl uses pkcs1 as it's default encoding rather than oaep. When I changed that, I got it to work.

Here is my code

    byte[] cipherText = Convert.FromBase64String(File.ReadAllText("./crypt.bin"));
    PemReader pr = new PemReader((StreamReader)File.OpenText("./privkey.pem"));
    var keys = (RsaKeyParameters)pr.ReadObject();
    var eng = new Pkcs1Encoding(new RsaEngine());
    eng.Init(false, keys);

    var length = cipherText.Length;
    var blockSize = eng.GetInputBlockSize();
    var plainTextBytes = new List<byte>();
    for (int chunkPosition = 0; chunkPosition < length; chunkPosition += blockSize)
    {
        var chunkSize = Math.Min(blockSize, length - chunkPosition);
        plainTextBytes.AddRange(eng.ProcessBlock(cipherText, chunkPosition, chunkSize));
    }
    Console.WriteLine(Encoding.UTF8.GetString(plainTextBytes.ToArray()));

I ran the encrypted text through a base64 encoding on my Linux box to lessen the risk of any errors when transfering. My code uses .NET core 3.1 and Bouncy Castle Core 1.86.

One thing that struck me in your code is the LoadKeys method. It doesn't really load the keys. It sets up a reader that can load the keys. My hunch is that it won't work if you ask for the same key twice. Do you do that?