How do fread and malloc affect RAM utilization in encryption process?

Question

I am working on an application (Linux based) that uses AES-256 encryption and observed a significant increase in RAM usage during encryption. Initially, the process consumes about 3 MB of RAM (checked via /proc/[pid]/status -> VmRSS), but during encryption, the consumption spikes to 25 MB. Additionally, I have observed that sometimes the 25 MB of RAM used during the encryption process is not fully released even after the encryption is complete. Please note that there are no memory leaks, as confirmed by thorough checks.

After adding more logs, I identified a few potential pain points contributing to this memory usage:

1. Malloc and memset : These are used to allocate and initialize buffers.I am allocating the malloc of the size of source file
2. fread : utilizing this function call to copy source file to buffer.
3. EVP_EncryptUpdate : This is part of the OpenSSL API for performing encryption. I’m not sure if it internally allocates additional memory or if my usage pattern is has some issue.

Now I am not sure how to debug it and control the RAM consumption.

Sample code

file_to_operate = fopen(file_name,"rb");
if (file_to_operate == NULL) {
    LOG_E ("File cannot be opened.");
    break;
}
     
//Get file size
fseek(file_to_operate, 0, SEEK_END);
fsize = ftell(file_to_operate);
fseek(file_to_operate, 0, SEEK_SET);  //same as rewind(f);

//Allocate buffer and read contents of file
buffer = (unsigned char* )malloc(fsize);
memset( buffer , 0 , fsize );
size_read = fread(buffer, fsize, 1, file_to_operate);
if (size_read < 1) { 
    LOG_E ("Buffer size read is invalid %d ", size_read);
    break;
}

op_buffer_padded_size = buffer_len + (MAX_PADDING - (buffer_len % MAX_PADDING));
op_file_size = op_buffer_padded_size + sizeof(op_file_t);
/* op_file_t is a structure 
 * uint64_t index;
 * uint64_t fsize;
 * uint64_t op_buffer_size;
*/

buffer_out = (op_file_t* )malloc(op_file_size);
if(NULL == buffer_out){
    LOG_E("malloc failed");
    break;
}

EVP_CIPHER_CTX *ctx = EVP_CIPHER_CTX_new();
unsigned char *output_buffer = (unsigned char *)malloc(file_size + AES_BLOCK_SIZE);
int output_len = 0;

EVP_CIPHER_CTX *ctx = EVP_CIPHER_CTX_new();
unsigned char *output_buffer = (unsigned char *)malloc(file_size + AES_BLOCK_SIZE);
int output_len = 0;

EVP_EncryptInit_ex(ctx, EVP_aes_256_cbc(), NULL, key, iv);
EVP_EncryptUpdate(ctx, output_buffer, &output_len, input_buffer, file_size);
EVP_EncryptFinal_ex(ctx, output_buffer + output_len, &output_len);

free(input_buffer);

//before freeing output buffer, write it to destination file.
free(output_buffer);
EVP_CIPHER_CTX_free(ctx);

What I Tried:

1. Debugging memory allocation:

• Added logging around malloc and memset to track the sizes of buffers being allocated.
• I expected these allocations to stay within reasonable limits based on the file size, but the overall memory usage still spiked unexpectedly.

2. Replace malloc + memset with calloc

3. Reading in smaller chunks:

• Tried modifying the fread logic to read the file in chunks (e.g., 4 KB or 8 KB) instead of allocating a single large buffer for the entire file.
• I expected this to reduce the peak RAM usage, but the memory still increased significantly during encryption.