StreamTransformationFilter: invalid PKCS #7 block padding found using AES decryption

Question

I am trying to perform AES decryption using the crypto++ library. I have an encrypted file whose first 8 bytes are the filelength, subsequent 16 bytes are the initialization vector, and the remaining data is the data of interest. I also have a string representation of my key (which I hash using SHA256)

I get the following error when trying to perform AES decryption: StreamTransformationFilter: invalid PKCS #7 block padding found

I am using the following c++ code:

std::string keyStr = "my_key";
std::string infilePath = "my/file/path";


CryptoPP::SHA256 hash;
unsigned char digest[CryptoPP::SHA256::DIGESTSIZE];

hash.CalculateDigest( digest, reinterpret_cast<const unsigned char*>(&keyStr[0]), keyStr.length() );
auto key = CryptoPP::SecByteBlock(digest, CryptoPP::SHA256::DIGESTSIZE);

std::ifstream fin(infilePath, std::ifstream::binary);

// First 8 bytes is the file size
std::vector<char> fileSizeVec(8);
fin.read(fileSizeVec.data(), fileSizeVec.size());

// Read the next 16 bytes to get the initialization vector
std::vector<char> ivBuffer(16);
fin.read(ivBuffer.data(), ivBuffer.size());
CryptoPP::SecByteBlock iv(reinterpret_cast<const unsigned char*>(ivBuffer.data()), ivBuffer.size());

// Create a CBC decryptor
CryptoPP::CBC_Mode<CryptoPP::AES>::Decryption decryption;
decryption.SetKeyWithIV(key, sizeof(key), iv);

CryptoPP::StreamTransformationFilter decryptor(decryption);

std::vector<char> buffer(CHUNK_SIZE, 0);

while(fin.read(buffer.data(), buffer.size())) {
    CryptoPP::SecByteBlock tmp(reinterpret_cast<const unsigned char*>(buffer.data()), buffer.size());
    decryptor.Put(tmp, tmp.size());
        }

decryptor.MessageEnd();
size_t retSize = decryptor.MaxRetrievable();

std::vector<char> decryptedBuff;
decryptedBuff.resize(retSize);

decryptor.Get(reinterpret_cast<CryptoPP::byte*>(decryptedBuff.data()), decryptedBuff.size());

I am not sure what is giving me the error. I am working off the following python code. When I run the python code with the same input file, it successfully decrypts the file.

def decrypt_file(in_filename, out_filename=None):
    key = hashlib.sha256(PASSWORD).digest()
    """loads and returns the embedded model"""
    chunksize = 24 * 1024
    if not out_filename:
        out_filename = os.path.splitext(in_filename)[0]

    with open(in_filename, 'rb') as infile:
        # get the initial 8 bytes with file size
        tmp = infile.read(8)
        iv = infile.read(16)
        decryptor = AES.new(key, AES.MODE_CBC, iv)
        string = b''
        # with open(out_filename, 'wb') as outfile:
        while True:
            chunk = infile.read(chunksize)
            if len(chunk) == 0:
                break
            string += decryptor.decrypt(chunk)
    return string

In addition to solving the error, I would also love some general c++ coding feedback on how I can improve.

Thanks in advance!

Edit: It looks like I wasn't reading the input file all the way to the end (as the length of the last chunk is smaller than CHUNK_SIZE). The following code now reads the entire file, however I still get the same issue. I have also confirmed that the IV and key match exactly that produced from the python code.

 // Get the length of the file in bytes
fin.seekg (0, fin.end);
size_t fileLen = fin.tellg();
fin.seekg (0, fin.beg);

std::vector<char> buffer(CHUNK_SIZE, 0);
size_t readSize = CHUNK_SIZE;

while(fin.read(buffer.data(), readSize)) {
     CryptoPP::SecByteBlock tmp(reinterpret_cast<const unsigned char*>(buffer.data()), CHUNK_SIZE);
     decryptor.Put(tmp, tmp.size());

      std::fill(buffer.begin(), buffer.end(), 0);

      size_t bytesReamining = fileLen - fin.tellg();
      readSize = CHUNK_SIZE < bytesReamining ? CHUNK_SIZE : bytesReamining;

       if (!readSize)
            break;
        }
}

Note that I have tried this line as both CryptoPP::SecByteBlock tmp(reinterpret_cast<const unsigned char*>(buffer.data()), CHUNK_SIZE); and CryptoPP::SecByteBlock tmp(reinterpret_cast<const unsigned char*>(buffer.data()), readSize); (Using CHUNK_SIZE pads with 0)

Answer 1

I have an encrypted file whose first 8 bytes are the filelength, subsequent 16 bytes are the initialization vector, and the remaining data is the data of interest...

I think I'll just cut to the chase and show you an easier way to do things with the Crypto++ library. The key and iv are hard-coded to simplify the code. The derivation is not needed for the example. By the way, if Python has it, you should consider using HKDF for derivation of the AES key and iv. HKDF has provable security properties.

Crypto++ handles the chunking for you. You don't need to explicitly perform it; see Pumping Data on the Crypto++ wiki.

I believe the Python code has a potential padding oracle present due to the use of CBC mode without a MAC. You might consider adding a MAC or using an Authenticated Encryption mode of operation.

#include "cryptlib.h"
#include "filters.h"
#include "osrng.h"
#include "modes.h"
#include "files.h"
#include "aes.h"
#include "hex.h"

#include <string>
#include <iostream>

const std::string infilePath = "test.dat";

int main(int argc, char* argv[])
{
    using namespace CryptoPP;

    const byte key[16] = {
        1,2,3,4, 1,2,3,4, 1,2,3,4, 1,2,3,4
    };
    const byte iv[16] = {
        8,7,6,5, 8,7,6,5, 8,7,6,5, 8,7,6,5
    };
    const byte data[] = // 70 characters
        "Now is the time for all good men to come to the aide of their country.";

    HexEncoder encoder(new FileSink(std::cout));
    std::string message;

    // Show parameters
    {        
        std::cout << "Key: ";
        StringSource(key, 16, true, new Redirector(encoder));
        std::cout << std::endl;

        std::cout << "IV: ";
        StringSource(iv, 16, true, new Redirector(encoder));
        std::cout << std::endl;

        std::cout << "Data: ";
        StringSource(data, 70, true, new Redirector(encoder));
        std::cout << std::endl;
    }

    // Write sample data
    {
        FileSink outFile(infilePath.c_str());
        word64 length = 8+16+70;

        outFile.PutWord64(length, BIG_ENDIAN_ORDER);
        outFile.Put(iv, 16);

        CBC_Mode<AES>::Encryption enc;
        enc.SetKeyWithIV(key, 16, iv, 16);

        StringSource(data, 70, true, new StreamTransformationFilter(enc, new Redirector(outFile)));
    }

    // Read sample data
    {
        FileSource inFile(infilePath.c_str(), true /*pumpAll*/);

        word64 read, l;
        read = inFile.GetWord64(l, BIG_ENDIAN_ORDER);
        if (read != 8)
            throw std::runtime_error("Failed to read length");

        SecByteBlock v(16);
        read = inFile.Get(v, 16);
        if (read != 16)
            throw std::runtime_error("Failed to read iv");

        CBC_Mode<AES>::Decryption dec;
        dec.SetKeyWithIV(key, 16, v, 16);

        SecByteBlock d(l-8-16);
        StreamTransformationFilter f(dec, new ArraySink(d, d.size()));
        inFile.CopyTo(f);
        f.MessageEnd();

        std::cout << "Key: ";
        StringSource(key, 16, true, new Redirector(encoder));
        std::cout << std::endl;

        std::cout << "IV: ";
        StringSource(v, 16, true, new Redirector(encoder));
        std::cout << std::endl;

        std::cout << "Data: ";
        StringSource(d, d.size(), true, new Redirector(encoder));
        std::cout << std::endl;

        message.assign(reinterpret_cast<const char*>(d.data()), d.size());
    }

    std::cout << "Message: ";
    std::cout << message << std::endl;

    return 0;
}

Running the program results in:

$ g++ test.cxx ./libcryptopp.a -o test.exe
$ ./test.exe
Key: 01020304010203040102030401020304
IV: 08070605080706050807060508070605
Data: 4E6F77206973207468652074696D6520666F7220616C6C20676F6F64206D656E20746F2063
6F6D6520746F207468652061696465206F6620746865697220636F756E7472792E

Key: 01020304010203040102030401020304
IV: 08070605080706050807060508070605
Data: 4E6F77206973207468652074696D6520666F7220616C6C20676F6F64206D656E20746F2063
6F6D6520746F207468652061696465206F6620746865697220636F756E7472792E

Message: Now is the time for all good men to come to the aide of their country.

---

Prior to this Stack Overflow question, the Crypto++ library did not provide PutWord64 and GetWord64. Interop with libraries like Python is important to the project, so they were added at Commit 6d69043403a9 and Commit 8260dd1e81c3. They will be part of the Crypto++ 8.3 release.

If you are working with Crypto++ 8.2 or below, you can perform the 64-bit read with the following code.

word64 length;
word32 h, l;

inFile.GetWord32(h, BIG_ENDIAN_ORDER);
inFile.GetWord32(l, BIG_ENDIAN_ORDER);

length = ((word64)h << 32) | l;

---

Here is the data file used for this example.

$ hexdump -C test.dat
00000000  00 00 00 00 00 00 00 5e  08 07 06 05 08 07 06 05  |.......^........|
00000010  08 07 06 05 08 07 06 05  b0 82 79 ee a6 d8 8a 0e  |..........y.....|
00000020  a6 b3 a4 7e 63 bd 9a bc  0e e4 b6 be 3e eb 36 64  |...~c.......>.6d|
00000030  72 cd ba 91 8d e0 d3 c5  cd 64 ae c0 51 de a7 c9  |r........d..Q...|
00000040  1e a8 81 6d c0 d5 42 2a  17 5a 19 62 1e 9c ab fd  |...m..B*.Z.b....|
00000050  21 3d b0 8f e2 b3 7a d4  08 8d ec 00 e0 1e 5e 78  |!=....z.......^x|
00000060  56 6d f5 3e 8c 5f fe 54                           |Vm.>._.T|

Answer 2

Looks like the issue had to do with padding. I instead switched to using a StringSource, which only worked once I specified CryptoPP::BlockPaddingSchemeDef::BlockPaddingScheme::ZEROS_PADDING as an argument for StreamTransformationFilter

Here is the working code for anyone that is interested:

    void Crypto::decryptFileAES(CryptoPP::SecByteBlock key, std::string infilePath) {
        std::ifstream fin(infilePath, std::ifstream::binary);

        // Get the length of the file in bytes
        fin.seekg (0, fin.end);
        size_t fileLen = fin.tellg();
        fin.seekg (0, fin.beg);

        // First 8 bytes is the file size
        std::vector<char> fileSizeVec(8);
        fin.read(fileSizeVec.data(), fileSizeVec.size());

        // Read the first 16 bytes to get the initialization vector
        std::vector<char> ivBuffer(16);
        fin.read(ivBuffer.data(), ivBuffer.size());
        CryptoPP::SecByteBlock iv(reinterpret_cast<const unsigned char*>(ivBuffer.data()), ivBuffer.size());

        // Create a CBC decryptor
        CryptoPP::CBC_Mode<CryptoPP::AES>::Decryption decryption;
        decryption.SetKeyWithIV(key, sizeof(key), iv);

        size_t bytesReamining = fileLen - fin.tellg();
        std::vector<char> buffer(bytesReamining);

        if(!fin.read(buffer.data(), bytesReamining)) {
            throw std::runtime_error("Unable to read file");
        }

        std::string decryptedText;

        CryptoPP::StringSource ss(reinterpret_cast<const unsigned char*>(buffer.data()), buffer.size(), true,
                new CryptoPP::StreamTransformationFilter(decryption,
                        new CryptoPP::StringSink(decryptedText), CryptoPP::BlockPaddingSchemeDef::BlockPaddingScheme::ZEROS_PADDING));

        std::cout << decryptedText << std::endl;
    }