CODEWITHSUNDEEP
c++encryptionopensslaes
CatalinC
CatalinC

Reputation: 77

OpenSSL AES encryption doesn't seem to work properly

So this is my very first time dabbling with AES encryption and OpenSSL. I have managed to get some examples of encryption and decryption, but they don't seem to work properly. For one, in the encryption function:

void encryptAES(FILE *iF, FILE *oF)
{
    fseek(iF, 0L, SEEK_END);
    int fsize = ftell(iF);
    fseek(iF, 0L, SEEK_SET);

    int out = 0; 
    int out2 = 0;
    unsigned char *inD = ( unsigned char * )malloc(fsize);
    unsigned char *outD = ( unsigned char * )malloc(fsize*2);
    unsigned char ckey[] =  "thiskeyisverybad";
    unsigned char ivec[] = "dontusethisinput";

    fread(inD,sizeof(char),fsize, iF);

    EVP_CIPHER_CTX* ctx = EVP_CIPHER_CTX_new();

    if(!EVP_EncryptInit(ctx,EVP_aes_128_cbc(),ckey,ivec))
    {
        std::cout<<"EVP_EncryptInit() error: "<<ERR_error_string(ERR_get_error(), NULL);
    }

    if(!EVP_EncryptUpdate(ctx,outD,&out,inD,fsize))
    {
        std::cout<<"EVP_EncryptUpdate() error: "<<ERR_error_string(ERR_get_error(), NULL);
    }

    if(!EVP_EncryptFinal(ctx,outD,&out2))
    {
        std::cout<<"EVP_EncryptFinal() error: "<<ERR_error_string(ERR_get_error(), NULL);
    }

    fwrite(outD,sizeof(char),fsize,oF);
}

This generates an actual "encrypted" file which is not half bad but it's not exactly what it's supposed to be (or I think so). If I run the cli command from openssl to encrypt that same file, I get in return a text file which has encrypted strings written inside. The one made by my function is a file which is not text (actually it says it is unkown format).

The second one, the decryption:

void decryptAES(FILE *iF, FILE *oF)
{
    fseek(iF, 0L, SEEK_END);
    int fsize = ftell(iF);
    fseek(iF, 0L, SEEK_SET);

    int out = 0;
    int out2 = 0;
    unsigned char *inD = ( unsigned char * )malloc(fsize);
    unsigned char *outD = ( unsigned char * )malloc(fsize*2);
    unsigned char ckey[] =  "thiskeyisverybad";
    unsigned char ivec[] = "djntusethisinput";

    fread(inD,sizeof(char),fsize, iF);//Read Entire File

    EVP_CIPHER_CTX * ctx = EVP_CIPHER_CTX_new();

    if(!EVP_DecryptInit(ctx,EVP_aes_128_cbc(),ckey,ivec))
    {
        std::cout<<"EVP_DecryptInit() error: "<<ERR_error_string(ERR_get_error(), NULL);
    }

    if(!EVP_DecryptUpdate(ctx,outD,&out,inD,fsize))
    {
        std::cout<<"EVP_DecryptUpdate() error: "<<ERR_error_string(ERR_get_error(), NULL);
    }

    if(!EVP_DecryptFinal(ctx, outD, &out2))
    {
        std::cout<<"EVP_DecryptFinal() error: "<<ERR_error_string(ERR_get_error(), NULL);
    }
    fwrite(outD,sizeof(char),fsize,oF);
}

In this function I actually get an error: EVP_DecryptFinal() error: error:0606506D:digital envelope routines:EVP_DecryptFinal_ex:wrong final block lengthP

Now, not sure if the problem is just in decryption because of the error, or also encryption.

Also, my main:

// Encrypt

FILE *ifp = fopen("Test.zip", "rb");
FILE *ofp = fopen("Test.zip.enc", "wb");

encryptAES(ifp, ofp);

fclose(ifp);
fclose(ofp);

// Decrypt

FILE *iF = fopen("Test.zip.enc", "rb");
FILE *oF = fopen("TEST.zip", "wb");

decryptAES(ifP, ofP);

fclose(iF);
fclose(oF);

Also, strangely enough, if my file is strictly a text file, it works (even though it should normally encrypt any type of file).

Upvotes: 3

Views: 373

Answers (1)

dbush
dbush

Reputation: 223689

Your calls to EVP_EncryptFinal and EVP_DecryptFinal are not writing to the correct place. Both write to outD, which is the beginning of the buffer. These need to write to the end of the buffer, i.e. out bytes later:

if(!EVP_EncryptFinal(ctx,outD+out,&out2))
...
if(!EVP_DecryptFinal(ctx,outD+out,&out2))

You're also not writing the correct number of bytes to the output files in both cases. The encrypted size is typically longer than the input size due to padding, but you're writing fsize bytes, which is the size of the input file. You want the number of bytes written, i.e. out+out2:

fwrite(outD,sizeof(char),out+out2,oF);
...
fwrite(outD,sizeof(char),out+out2,oF);

There's also an IV mismatch. For encryption you use "dontusethisinput" while for decryption you use "djntusethisinput" (note the second letter in each). These need to be the same.

You should also check the return values of the functions doing file I/O in case they don't work as expected, and also free any dynamically allocated memory.

Upvotes: 1

Related Questions