Abort trap 6 when returning from main in OS X but NOT on linux

Question

I have a program that currently seems to run fine on Linux (Ubuntu 14.04), but when running on OS X (10.11.6) I get an abort trap 6. I've attached my code but I suspect the problem is not actually tied to the specific code. This code is for a class project, I'm not actually trying to crack passwords or anything.

Here's the code, I believe the all the important stuff happens in main.

#include <openssl/aes.h>   
#include <openssl/evp.h>
#include <openssl/conf.h>
#include <openssl/err.h>


#define KEY_BYTES KEY_LENGTH/8
#define KEY_LENGTH 128


unsigned char*  h(unsigned char* p, unsigned char* hp);
void handleErrors(void);
int encrypt(unsigned char *plaintext, int plaintext_len, unsigned char *key,
            unsigned char *iv, unsigned char *ciphertext);


//assumes we've already padded with zeros
unsigned char*  h(unsigned char* p, unsigned char *hp){
    encrypt((unsigned char*)"0000000000000000", KEY_BYTES, p , (unsigned char*)"0000000000000000", hp);
    return hp; 
}

void handleErrors(void)
{
    printf("panic!!\n");
    //ERR_print_errors_fp(stderr); //sg: throw a real error you fool!
    abort();
}

//sg: stolen from the evp man page
int encrypt(unsigned char *plaintext, int plaintext_len, unsigned char *key,
            unsigned char *iv, unsigned char *ciphertext)
{
    EVP_CIPHER_CTX *ctx;

    int len;

    int ciphertext_len;

    /* Create and initialise the context */
    if(!(ctx = EVP_CIPHER_CTX_new())) handleErrors();

    /* Initialise the encryption operation. IMPORTANT - ensure you use a key
     * and IV size appropriate for your cipher
     * In this example we are using 256 bit AES (i.e. a 256 bit key). The
     * IV size for *most* modes is the same as the block size. For AES this
     * is 128 bits */
    if(1 != EVP_EncryptInit_ex(ctx, EVP_aes_128_cbc(), NULL, key, iv))
        handleErrors();

    /* Provide the message to be encrypted, and obtain the encrypted output.
     * EVP_EncryptUpdate can be called multiple times if necessary
     */
    if(1 != EVP_EncryptUpdate(ctx, ciphertext, &len, plaintext, plaintext_len))
        handleErrors();
    ciphertext_len = len;

    /* Finalise the encryption. Further ciphertext bytes may be written at
     * this stage.
     */
    if(1 != EVP_EncryptFinal_ex(ctx, ciphertext + len, &len)) handleErrors();
    ciphertext_len += len;

    /* Clean up */
    EVP_CIPHER_CTX_free(ctx);

    return ciphertext_len;
}


int main(){

    /* Initialise the library */
    ERR_load_crypto_strings();
    OpenSSL_add_all_algorithms();
    OPENSSL_config(NULL);

    EVP_CIPHER_CTX *ctx;
    unsigned char hp[KEY_BYTES];

    /* Create and initialise the context */
    if(!(ctx = EVP_CIPHER_CTX_new())) handleErrors();

    h((unsigned char*) "1111111111111111", hp);

    for(int i = 0; i < KEY_BYTES; i++){
        printf("h(%i) = %x\n", i, hp[i]);
    }

    return 0;
}

When run on linux I get the following (which is what I expect)

h(0) = 10
h(1) = df
h(2) = c1
h(3) = b5
h(4) = f6
h(5) = 6c
h(6) = fd
h(7) = 6a
h(8) = 1d
h(9) = c4
h(10) = 6d
h(11) = 66
h(12) = 90
h(13) = 7b
h(14) = ee
h(15) = b1

However when I run on OS X I get the following:

h(0) = 10
h(1) = df
h(2) = c1
h(3) = b5
h(4) = f6
h(5) = 6c
h(6) = fd
h(7) = 6a
h(8) = 1d
h(9) = c4
h(10) = 6d
h(11) = 66
h(12) = 90
h(13) = 7b
h(14) = ee
h(15) = b1
Abort trap: 6

When I pop this into gdb I get the following

(gdb) r
Starting program: /Users/sgillen/Code/457/proj3/a.out 
h(0) = 10
h(1) = df
h(2) = c1
h(3) = b5
h(4) = f6
h(5) = 6c
h(6) = fd
h(7) = 6a
h(8) = 1d
h(9) = c4
h(10) = 6d
h(11) = 66
h(12) = 90
h(13) = 7b
h(14) = ee
h(15) = b1

Program received signal SIGABRT, Aborted.
0x00007fff93150f06 in __pthread_kill () from /usr/lib/system/libsystem_kernel.dylib
(gdb) where
#0  0x00007fff93150f06 in __pthread_kill () from /usr/lib/system/libsystem_kernel.dylib
#1  0x00007fff97b374ec in pthread_kill () from /usr/lib/system/libsystem_pthread.dylib
#2  0x00007fff9ba8077f in __abort () from /usr/lib/system/libsystem_c.dylib
#3  0x00007fff9ba8105e in __stack_chk_fail () from /usr/lib/system/libsystem_c.dylib
#4  0x0000000100000ea9 in main () at gen_table.cpp:90

Not sure how to do line numbers on stack overflow, but line 90 of gen_table.cpp is that last return 0 in main.

I compile my code with the following if that's relevant.

clang -Wall -std=c++11 gen_table.cpp  -I/usr/local/opt/openssl/include/ -lcrypto -lssl -g

Any help would be greatly appreciated thank you!

Answer 1

I found the answer to my question I figured I'd post the answer for anyone else who somehow runs into the same problem. The issue was that I was overwriting my own stack. The encryption function I was using was actually writing 32 bytes to hp (which was a 16 byte unsigned char living on the stack). So I'd destroy my own stack but not write to any memory not owned by my process. This resulted in no seg faults but when the program tried to return there were eventually problems. The exact thing that killed me changed depending on how I compiled my code.

I'm actually very surprised valgrind didn't catch this. And I still don't know why it seemed to work fine on linux when compiled with clang (compiled with g++ I got a stack-smashing detected error).

edit: To be clear the solution was to fix my implementation of encrypt so that it only writes 16 bytes. which I simply did by commenting out the EVP_EncryptFinal_ex call.