Error while decrypting file in java which is encrypted in.Net using Rijndael

Question

I am given a Rijndael .Net encrypted file and .Net RSA XML Key and asked to decrypt it in Java.

The key provided to me is 256 bit.

I have parsed the RSA XML file and generated the public Key in Java. I tried to decrypt using the generated key, however I am getting the exception Illegal Key Size, I think I am doing something wrong in my Java code.

Can any one please help to check if anything is wrong with my code?

.Net encryption code:

public static void EncryptFile(string fileIn, string fileOut,
                                   string publicKeyName, string publicKeyFile)
    {
        try
        {
            // Read the public key from key file
            StreamReader sr = new StreamReader(publicKeyFile);
            string strKeyText = sr.ReadToEnd();
            sr.Close();

            //Initialize Key container and Crypto service provider
            RSACryptoServiceProvider rsa;
            CspParameters cspp = new CspParameters();
            cspp.KeyContainerName = publicKeyName;
            rsa = new RSACryptoServiceProvider(cspp);
            rsa.FromXmlString(strKeyText);
            rsa.PersistKeyInCsp = true;

            // Create instance of Rijndael for
            // symetric encryption of the data.
            RijndaelManaged alg = new RijndaelManaged();
            // Key size is set to 256 for strong encryption
            alg.KeySize = 256;
            alg.BlockSize = 256;
            // Cipher Mode is set to CBC to process the file in chunks
            alg.Mode = CipherMode.CBC;
            // Set padding mode to process the last block of the file
            alg.Padding = PaddingMode.ISO10126;

            ICryptoTransform transform = alg.CreateEncryptor();

            // Use RSACryptoServiceProvider to
            // enrypt the Rijndael key.
            byte[] KeyEncrypted = rsa.Encrypt(alg.Key, false);

            // Create byte arrays to contain
            // the length values of the key and IV.
            int intKeyLength = KeyEncrypted.Length;
            byte[] LenK = BitConverter.GetBytes(intKeyLength);
            int intIVLength = alg.IV.Length;
            byte[] LenIV = BitConverter.GetBytes(intIVLength);


            using (FileStream fsOut = new FileStream(fileOut, FileMode.Create))
            {
                // Write the following to the FileStream
                // for the encrypted file (fsOut):
                // - length of the key
                // - length of the IV
                // - ecrypted key
                // - the IV
                // - the encrypted cipher content
                fsOut.Write(LenK, 0, 4);
                fsOut.Write(LenIV, 0, 4);
                fsOut.Write(KeyEncrypted, 0, intKeyLength);
                fsOut.Write(alg.IV, 0, intIVLength);

                // Now write the cipher text using
                // a CryptoStream for encrypting.
                using (CryptoStream cs = new CryptoStream(fsOut, transform, CryptoStreamMode.Write))
                {
                    // intBlockSizeBytes can be any arbitrary size.
                    int intBlockSizeBytes = alg.BlockSize / 8;
                    byte[] DataBytes = new byte[intBlockSizeBytes];
                    int intBytesRead = 0;

                    using (FileStream fsIn = new FileStream(fileIn, FileMode.Open))
                    {
                        // By encrypting a chunk at
                        // a time, you can save memory
                        // and accommodate large files.
                        int intCount;
                        int intOffset = 0;

                        do
                        {
                            // if last block size is less than encryption chunk size
                            // use the last block size and padding character is used 
                            // for remaining bytes
                            if (intBlockSizeBytes > (fsIn.Length - fsIn.Position))
                            {
                                intBlockSizeBytes = ((int)(fsIn.Length - fsIn.Position));
                                DataBytes = new byte[intBlockSizeBytes];
                            }
                            // read data bytes
                            intCount = fsIn.Read(DataBytes, 0, intBlockSizeBytes);
                            intOffset += intCount;
                            // write it into crypto stream
                            cs.Write(DataBytes, 0, intCount);
                            intBytesRead += intBlockSizeBytes;
                        } while (intCount > 0);
                        // close input file
                        fsIn.Close();
                    }
                    // close crypto stream
                    cs.FlushFinalBlock();
                    cs.Close();
                }
                // close output file
                fsOut.Close();
            }
        }
        catch
        {
            throw;
        }
    }

Java Code that I wrote to decrypt it:

    byte[] expBytes = Base64.decodeBase64(pkey.getExponentEle().trim());
    byte[] modBytes = Base64.decodeBase64(pkey.getModulusEle().trim());
    byte[] dBytes = Base64.decodeBase64(pkey.getdEle().trim());

    BigInteger modules = new BigInteger(1, modBytes);
    BigInteger exponent = new BigInteger(1, expBytes);
    BigInteger d = new BigInteger(1, dBytes);

    KeyFactory factory = KeyFactory.getInstance("RSA");
    Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");

    RSAPublicKeySpec pubSpec = new RSAPublicKeySpec(modules, exponent);
    PublicKey pubKey = factory.generatePublic(pubSpec);

    final byte[] keyData = Arrays.copyOf(pubKey.getEncoded(), 256
            / Byte.SIZE);
        final byte[] ivBytes = Arrays.copyOf(keyData, cipher.getBlockSize());
        AlgorithmParameterSpec paramSpec = new IvParameterSpec(ivBytes);

        cipher.init(Cipher.DECRYPT_MODE, new SecretKeySpec(keyData, "AES"), paramSpec);
    byte[] decrypted = cipher.doFinal(encrypted);
    System.out.println("decrypted: " + new String(decrypted));

If I change cipher initialization to cipher.init(Cipher.DECRYPT_MODE, pubKey);, then I am getting the error Invalid AES key length: 162 bytes

Answer 1

You are using the public key wrong way. Do you really understad how the C# program works? what parameters is it using?

You are just using the public key bits as the AES key (even I don't realy understand how do you get 162 bytes from it).

This is example of "hybrid encryption" - the data themselves are encrypted by a random AES key (in this you claim it's 256 bit) and the AES key (in this case the IV too) is encrypted by the RSA public key. In Java there are many examples how to do that.

Even to decrypt the AES key you should know parameters used to encrypt it (RSA/ECB/PKCS5Padding, RSA-AOEP, ...), though it should be inside the XML.

Comming to the parameters - you are using PKCS5Padding, but check the .NET code, it's different