Why is RSA encryption in Java producing different length ciphertext than C#

Question

I am AES encrypting some text using a randomly generated key and then RSA encrypting that key with a private key so that I can upload it to a database.

The RSA keys are generated using KeyPairGenerator in Java and saved as a file. Keys are read in using File.ReadAllBytes().

When I do this in Java everything works perfectly and the encrypted key is always 172 bytes, but when I do it in C# the encrypted key is always 844 bytes. I'm pretty sure the text is being encrypted properly using AES, but something is going wrong with the RSA encryption.

I have checked the key sizes in Java and C# and they always match. Literally, the only difference I can see is the RSA encrypted ciphertext length, which makes the data unusable. I believe it has something to do with padding, but I don't know how to fix it.

Java

public String encryptText(String msg, PrivateKey key) 
            throws NoSuchAlgorithmException, NoSuchPaddingException,
            UnsupportedEncodingException, IllegalBlockSizeException, 
            BadPaddingException, InvalidKeyException {

        KeyGenerator generator;   
        this.cipher.init(Cipher.ENCRYPT_MODE, key); //cipher is initialized earlier with this.cipher = Cipher.getInstance("RSA");

            try {
                generator = KeyGenerator.getInstance(AES);
                generator.init(128); // The AES key size in number of bits
                SecretKey secKey = generator.generateKey();

                Cipher aesCipher = Cipher.getInstance(AES);
                aesCipher.init(Cipher.ENCRYPT_MODE, secKey);

                String encText = Base64.getEncoder().encodeToString(aesCipher.doFinal(msg.getBytes("UTF-8")));
                String encKey = Base64.getEncoder().encodeToString(cipher.doFinal(secKey.getEncoded()));

                return "(" + encText + ")" + encKey;                
            } catch (NoSuchAlgorithmException | NoSuchPaddingException | InvalidKeyException | IllegalBlockSizeException | BadPaddingException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }

        return null;
    }

C#

public String EncryptText(byte[] privateKeyBytes, string msg)
        {
            try
            {
                RSACryptoServiceProvider RSA = new RSACryptoServiceProvider();
                RSAParameters RSAKeyInfo = RSA.ExportParameters(false);

                RSAKeyInfo.Modulus = privateKeyBytes;
                RSA.ImportParameters(RSAKeyInfo);

                RijndaelManaged aes = new RijndaelManaged();
                aes.BlockSize = 128;
                aes.KeySize = 128;
                aes.Mode = CipherMode.ECB;
                byte[] keyGenerated = aes.Key;

                string keyStr = Convert.ToBase64String(keyGenerated);
                byte[] keyArr = Convert.FromBase64String(keyStr);
                byte[] KeyArrBytes16Value = new byte[16];
                Array.Copy(keyArr, KeyArrBytes16Value, 16);

                aes.Key = KeyArrBytes16Value;

                ICryptoTransform encrypto = aes.CreateEncryptor();

                byte[] plainTextByte = ASCIIEncoding.UTF8.GetBytes(msg);
                byte[] CipherText = encrypto.TransformFinalBlock(plainTextByte, 0, plainTextByte.Length);

                string encText = Convert.ToBase64String(CipherText);
                string encKey = Convert.ToBase64String(RSA.Encrypt(aes.Key, true));

                return "(" + encText + ")" + encKey;

            }
            catch (CryptographicException e)
            {
                Console.WriteLine("FAILED: " + e.Message);
            }

            return null;
        }

UPDATE
Thanks to Henno for pointing out that the problem was in how I was reading the key. I ended up using Bouncy Castle to handle the RSA encryption in C#. I also changed my java code to encrypt with the public key instead of the private key.

New C#

public String EncryptText(byte[] keyBytes, string msg)
        {
            try
            {
                AsymmetricKeyParameter asymmetricKeyParameter = PublicKeyFactory.CreateKey(keyBytes);
                RsaKeyParameters rsaKeyParameters = (RsaKeyParameters)asymmetricKeyParameter;
                RSAParameters rsaParameters = new RSAParameters();
                rsaParameters.Modulus = rsaKeyParameters.Modulus.ToByteArrayUnsigned();
                rsaParameters.Exponent = rsaKeyParameters.Exponent.ToByteArrayUnsigned();
                RSACryptoServiceProvider rsa = new RSACryptoServiceProvider();
                rsa.ImportParameters(rsaParameters);

                RijndaelManaged aes = new RijndaelManaged();
                aes.BlockSize = 128;
                aes.KeySize = 128;
                aes.Mode = CipherMode.ECB;
                byte[] keyGenerated = aes.Key;

                string keyStr = Convert.ToBase64String(keyGenerated);
                byte[] keyArr = Convert.FromBase64String(keyStr);
                byte[] KeyArrBytes16Value = new byte[16];
                Array.Copy(keyArr, KeyArrBytes16Value, 16);

                aes.Key = KeyArrBytes16Value;

                ICryptoTransform encrypto = aes.CreateEncryptor();

                byte[] plainTextByte = ASCIIEncoding.UTF8.GetBytes(msg);
                byte[] CipherText = encrypto.TransformFinalBlock(plainTextByte, 0, plainTextByte.Length);

                string encText = Convert.ToBase64String(CipherText);
                string encKey = Convert.ToBase64String(rsa.Encrypt(aes.Key, false));

                return "(" + encText + ")" + encKey;

            }
            catch (CryptographicException e)
            {
                Console.WriteLine("FAILED: " + e.Message);
            }

            return null;
        }

Answer 1

What seems to go wrong is that you read in the saved "private key file" in C#, presumably in the variable privateKeyBytes (but your code is incomplete, so I'm guessing) and then do RSAKeyInfo.Modulus = privateKeyBytes, which is weird and cryptographically implausible. You should instantiate some kind of RSA class in C# as well, based on the bytes you read in, which is what I think you're trying to do in the beginning of the C# code (first four lines). I think there should be another API for that, looking around in the docs:

RSA.ImportParameters(RSAKeyInfo) and then maybe set RSAKeyInfo from those bytes, but it's not the modulus. The read in bytes should be PKCS1 format or something similar, maye base64 encoded in file, or raw etc. You'd have to look into what format Java uses to export full keys to disk.

You use the raw bytes you read in from file as a modulus, which is surely going to give trouble and gives a "key" that is invalid and much too big as well.