Encryption using Openssl EVP library and decryption using openssl enc terminal command fails

Question

I am trying to encrypt a string using OpenSSL EVP library, and decrypt subsequently using the openssl command from a linux terminal. OpenSSL command for decryption:

openssl enc -aes-256-cbc -base64 -salt -d -md md5 -k <passphrase> -in encrypt.txt -out plain.txt

Encryption code:

int CAES::encrypt(const char* msg, unsigned char** encrypted_message, const size_t msg_len, const unsigned char *key, unsigned char *iv, const unsigned char* salt) 
{
    //this buffer will hold the salt and the cipher
    const int buffer_len ( CAES::salt_info_size + std::max((int)(msg_len * 2), EVP_MAX_KEY_LENGTH) );
    unsigned char* buffer = new unsigned char[buffer_len];
    //this will be a pointer to the cipher only
    unsigned char* cipher = CAES::salt_info_size + buffer;
    size_t bytes_encrypted(0);
    bool encryption_error(true);
    EVP_CIPHER_CTX *cipher_ctx( create_cipher_ctx(encryption, key, iv) );
    (void) EVP_CIPHER_CTX_set_key_length(cipher_ctx, EVP_MAX_KEY_LENGTH);
        size_t cipher_len(0);
    int temp_len(0);
    if( EVP_EncryptUpdate(cipher_ctx, cipher, &temp_len, (unsigned char *)msg, strlen(msg)) )
    {
        cipher_len = temp_len;
        if( EVP_EncryptFinal_ex(cipher_ctx, cipher + temp_len, &temp_len) )
        {
            cipher_len += temp_len;
        }
        cipher[cipher_len] = '\0';
        encryption_error = false;
        bytes_encrypted = cipher_len;
    }
    memcpy(buffer + 0 , CAES::salt_tag, CAES::salt_tag_size);
    memcpy(buffer + CAES::salt_tag_size, salt, CAES::salt_size);
    memcpy(*encrypted_message, buffer, CAES::salt_info_size + bytes_encrypted);
    EVP_CIPHER_CTX_free(cipher_ctx);
    delete[] buffer;
    return CAES::salt_info_size + bytes_encrypted;
}

Code to encode the encrypted string from above to base64:

int CBase64::encode(const unsigned char* msg, const size_t msg_len, char** b64_msg) {
  size_t bytes_encoded = 0;
  bytes_encoded = EVP_EncodeBlock((unsigned char *) *b64_msg, msg, msg_len);
  return bytes_encoded;
}

Key and IV derivation

void CAES::init_key_iv(const std::string& pass, const unsigned char* salt, unsigned char* key, unsigned char* iv )
{
  const unsigned char * pass_key = reinterpret_cast<const unsigned char*>( pass.c_str() );
  const size_t pass_key_len ( pass.size() );
  EVP_BytesToKey(CAES::cipher_type, CAES::msg_digest_type, salt, pass_key, pass_key_len, 1, key, iv);
}

The following are the key sizes and salt derivation:

unsigned char salt[8] = {};
unsigned char key[32] = {};
unsigned char iv_enc[16] = {};
RAND_bytes(salt, 8);

Case 1:

const char* password = "@N";
const char* msg = "neo4j";

On trying to decrypt I get the following:

bad decrypt
140589946300064:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:evp_enc.c:539:

Case 2:

const char* password = "eemnsis";
const char* msg = "This is long message, repeating 10 times.This is long message, repeating 10 times.This is long message, repeating 10 times.This is long message, repeating 10 times.This is long message, repeating 10 times.This is long message, repeating 10 times.This is long message, repeating 10 times.This is long message, repeating 10 times.This is long message, repeating 10 times.This is long message, repeating 10 times.";

On trying to decrypt I get the following:

error reading input file

Answer 1

I suspect it's the Base64-encoding.

In the openssl statement for decryption, the -base64 option is used without the -A option. Therefore, a formatted Base64-string is expected, i.e. after each 64 byte and at the end there are line breaks, see -A-Option.

However, the current encode function does not do this, but writes the entire Base64-string in a single line.

The problem can be solved by

• either adding the -A option to the openssl statement for decryption: -base64 -A

• or modifying the encode method to include line breaks, e.g. with EVP_EncodeUpdate etc.):

  int CBase64::encode(const unsigned char* msg, const size_t msg_len, char** b64_msg) {
      int len, total = 0;
      EVP_ENCODE_CTX *ectx = EVP_ENCODE_CTX_new();
      EVP_EncodeInit(ectx);
      EVP_EncodeUpdate(ectx, (unsigned char *)*b64_msg, &len, msg, msg_len);
      total += len;
      EVP_EncodeFinal(ectx, (unsigned char *)*b64_msg + len, &len);
      total += len;
      EVP_ENCODE_CTX_free(ectx);
      return total;
  }