AES - simple encrypt in Java, decrypt with openssl

Question

I am trying to do a simple AES encryption in Java, using Java Cryto, that can then be decrypted in ObjectiveC, using OpenSSL.

as I am not doing the ObjectiveC side, I want to make sure it works, using the openSSL command line, but I always get "bad magic number"

Here is my Java code

public class EncryptionUtils {

private static final String AES_CIPHER_METHOD = "AES";
private static final int AES_KEY_SIZE = 128;

public static byte[] generateAesKey() throws NoSuchAlgorithmException {
    KeyGenerator keyGenerator = KeyGenerator.getInstance(AES_CIPHER_METHOD);
    keyGenerator.init(AES_KEY_SIZE);
    SecretKey key = keyGenerator.generateKey();
    return key.getEncoded();
}

public static SecretKeySpec createAesKeySpec(byte[] aesKey) {
    return new SecretKeySpec(aesKey, AES_CIPHER_METHOD);
}

public static void aesEncryptFile(File in, File out, SecretKeySpec aesKeySpec) throws InvalidKeyException, NoSuchAlgorithmException, NoSuchPaddingException, IOException {
    Cipher aesCipher = Cipher.getInstance(AES_CIPHER_METHOD);
    aesCipher.init(Cipher.ENCRYPT_MODE, aesKeySpec);
    InputStream inputStream = new FileInputStream(in);
    try {
        OutputStream outputStream = new CipherOutputStream(new FileOutputStream(out), aesCipher);
        try {
            IOUtils.copy(inputStream , outputStream);
        } finally {
            outputStream.close();
        }
    } finally {
        inputStream.close();
    }
}
}


//testcode
@Test
public void testAesEncryptFile() throws IOException, InvalidKeyException, IllegalBlockSizeException, BadPaddingException, NoSuchAlgorithmException, NoSuchPaddingException {
    byte[] aesKey = EncryptionUtils.generateAesKey();
    SecretKeySpec aesKeySpec = EncryptionUtils.createAesKeySpec(aesKey);
    EncryptionUtils.aesEncryptFile(new File("C:\\test\\test.txt"), new File("C:\\test\\test-encrypted.txt"), aesKeySpec);

    FileOutputStream outputStream = new FileOutputStream("C:\\test\\aes.key");
    outputStream.write(aesKey);
    outputStream.close();
}

@Test
public void testAesDecryptFile() throws IOException, InvalidKeyException, IllegalBlockSizeException, BadPaddingException, NoSuchAlgorithmException, NoSuchPaddingException {
    FileInputStream keyFis = new FileInputStream("C:\\test\\aes.key");
    ByteArrayOutputStream keyBaos = new ByteArrayOutputStream();
    IOUtils.copy(keyFis, keyBaos);

    SecretKeySpec keySpec = new SecretKeySpec(keyBaos.toByteArray(), "AES");
    Cipher cipher = Cipher.getInstance("AES");
    cipher.init(Cipher.DECRYPT_MODE, keySpec);

    FileInputStream fileInputStream = new FileInputStream("C:\\test\\test-encrypted.txt");
    ByteArrayOutputStream baos = new ByteArrayOutputStream();
    IOUtils.copy(fileInputStream, baos);

    byte[] decrypted = cipher.doFinal(baos.toByteArray());
    FileOutputStream outputStream = new FileOutputStream("C:\\test\\test-decrypted.txt");
    outputStream.write(decrypted);
    outputStream.close();

}

Now that runs as expected, file "test-encrypted.txt" is indeed encrypted, and "test-decrypted.txt" == "test.txt"

I then tried to run a decryption on the command line using OpenSSL

openssl enc -d -aes-128-ecb -in test-encrypted.txt -k aes.key

however, this always give me

bad magic number

From what I can see, the using algorithm "AES" in Java uses "ECB" mode by default, so the above should work. What am I doing wrong.

Answer 1

I was struggling with the same issue for a few days. My task was to encode file using AES/CBC/PKCS5PADDING and given password phrase in java so that it should be decodeable with command like follow:

.\openssl.exe enc -d -aes-256-cbc -in "test_enc.txt" -out "test_dec.txt" -iter 1000000

When I encoded the file using simple code (like this https://www.baeldung.com/java-cipher-input-output-stream) I kept getting 'bad magic number' or 'bad decrypt' after some modifications. Finally, basing on openssl code (https://github.com/openssl/openssl/blob/master/apps/enc.c) I've managed to write code that works. Below is the result:

import lombok.SneakyThrows;

import javax.crypto.Cipher;
import javax.crypto.CipherOutputStream;
import javax.crypto.SecretKey;
import javax.crypto.SecretKeyFactory;
import javax.crypto.spec.IvParameterSpec;
import javax.crypto.spec.PBEKeySpec;
import javax.crypto.spec.SecretKeySpec;
import java.io.OutputStream;
import java.security.SecureRandom;
import java.security.spec.KeySpec;
import java.util.Arrays;

import static java.nio.charset.StandardCharsets.US_ASCII;

public class EncryptionUtil {

    private static final int KEY_LENGTH = 32;
    private static final int IV_LENGTH = 16;

    /** OpenSSL's magic initial bytes. */
    private static final String SALTED_STR = "Salted__";
    private static final byte[] SALTED_MAGIC = SALTED_STR.getBytes(US_ASCII);

    @SneakyThrows
    public static OutputStream encodeOutputStream(OutputStream outputStream, String password) {
        Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5PADDING");

        byte[] salt = new SecureRandom().generateSeed(8);
        SecretKeyFactory factory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA256");
        KeySpec spec = new PBEKeySpec(password.toCharArray(), salt, 1000000, (KEY_LENGTH + IV_LENGTH)*8);
        byte[] encodedKey = factory.generateSecret(spec).getEncoded();
        SecretKey secretKey = new SecretKeySpec(Arrays.copyOfRange(encodedKey, 0, KEY_LENGTH), "AES");
        IvParameterSpec iv = new IvParameterSpec(Arrays.copyOfRange(encodedKey, KEY_LENGTH, encodedKey.length));

        outputStream.write(SALTED_MAGIC);
        outputStream.write(salt);

        cipher.init(Cipher.ENCRYPT_MODE, secretKey, iv);

        return new CipherOutputStream(outputStream, cipher);
    }
}

Above code we can simply use for example like this:

try (InputStream in = new FileInputStream(inFilePath);
     OutputStream encryptedOutputStream = EncryptionUtil.encodeOutputStream(new FileOutputStream(encFilePath), "password")) {
     IOUtils.copy(in, encryptedOutputStream);
}

Apart from source openssl code, it is also inspired with code presented in another topic: https://stackoverflow.com/a/41495143 but I suppose there is EVP_BytesToKey method (https://www.openssl.org/docs/man3.1/man3/EVP_BytesToKey.html) used (like in @Vic's answer) and I required PBKDF2 (https://www.openssl.org/docs/man3.0/man7/EVP_KDF-PBKDF2.html).

Answer 2

The problem is indeed due to the key that is computed from the password by OpenSSL.

Most likely the reason is that OpenSSL has its own algorithm to derive a key, EVP_BytesToKey, from the password, and that is not the same as Java's.

The only solution I found was to use a Java reimplementation of that algorithm:

private static final int KEY_LENGTH = 32;    

private byte[] deriveKey(String encryptionPassword, byte[] salt) throws NoSuchAlgorithmException {
    final byte[] passAndSalt = ArrayUtils.addAll(encryptionPassword.getBytes(), salt);
    byte[] hash = new byte[0];
    byte[] keyAndIv = new byte[0];
    for (int i = 0; i < 3 && keyAndIv.length < KEY_LENGTH; i++) {
        final byte[] dataToHash = ArrayUtils.addAll(hash, passAndSalt);
        final MessageDigest md = MessageDigest.getInstance("SHA-256");
        hash = md.digest(dataToHash);
        keyAndIv = ArrayUtils.addAll(keyAndIv, hash);
    }
    return Arrays.copyOfRange(keyAndIv, 0, KEY_LENGTH);
}

ArrayUtils is part of Apache Commons library.

Full usage:

IvParameterSpec initializationVectorSpec = new IvParameterSpec(
                Hex.decodeHex(encryptionInitializationVector.toCharArray()));

cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
byte[] salt = new SecureRandom().generateSeed(8);
byte[] key = deriveKey(encryptionPassword, salt);
Key keySpec = new SecretKeySpec(key, "AES");
cipher.init(Cipher.ENCRYPT_MODE, keySpec, initializationVectorSpec);

byte[] rawEncryptedInput = cipher.doFinal(input.getBytes());
byte[] encryptedInputWithPrependedSalt = ArrayUtils.addAll(ArrayUtils.addAll(
                "Salted__".getBytes(), salt), rawEncryptedInput);
return Base64.getEncoder()
                .encodeToString(encryptedInputWithPrependedSalt);

Credit to this answer for showing me the way.

Answer 3

The problem is with the key. The -k argument expects a passphrase, not a file. In turn, when a passphrase is used by the openssl encryption routine, a magic and salt is put in front of the encrypted result. That's the magic that cannot be found.

To use the openssl command line, print out the key in hex and use the -K option instead of the lowercase -k option.

You could also use:

`cat aes.key`

as argument after -K, given that aes.key contains the key in hexadecimals.