Getting Bad Request 400 Errors Trying to Upload File to Google Cloud Storage Using A Generated MD5 or CRC32C Checksum for the File

Question

I am trying to upload files to Google Cloud Storage using the Google Cloud Storage Java APIs and by setting either an md5 or crc32c checksum/hash for the file before uploading.

I keep getting errors similar to what is shown below (for both md5 and crc32c hashes) and have no idea what I am doing wrong:

{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "invalid",
    "message": "Provided CRC32C \"qtQiXA==\" doesn't match calculated CRC32C \"AAAAAA==\"."
   }
  ],
  "code": 400,
  "message": "Provided CRC32C \"qtQiXA==\" doesn't match calculated CRC32C \"AAAAAA==\"."
 }
}

The file upload works if no md5 or crc32c checksum values are set on the BlobInfo. The code snippet of what I am trying to achieve before using a WriteChannel to upload the file blob is shown below:

// Upload a blob to the identified bucket and use md5 or crc32c checksum 
BlobId blobId = BlobId.of(storageBucketName, sourceEvent.getFileName());
BlobInfo blobInfo = null;
Storage.BlobWriteOption blobWriteOption = null;

if(sourceEvent.getFileHash() != null && sourceEvent.getFileHashType() == FileIntegrityCheckType.CRC32C.getCheckType()) {
    blobInfo = BlobInfo.newBuilder(blobId).setStorageClass(
            GoogleCloudStorageClass.getStorageClass(storageClass)
    ).setCrc32c(sourceEvent.getFileHash()).build();
    blobWriteOption = Storage.BlobWriteOption.crc32cMatch();
} else if(sourceEvent.getFileHash() != null && sourceEvent.getFileHashType() == FileIntegrityCheckType.MD5.getCheckType()) {
    blobInfo = BlobInfo.newBuilder(blobId).setStorageClass(
            GoogleCloudStorageClass.getStorageClass(storageClass)
    ).setMd5(sourceEvent.getFileHash()).build();
    blobWriteOption = Storage.BlobWriteOption.md5Match();
} else {
    blobInfo = BlobInfo.newBuilder(blobId).setStorageClass(
            GoogleCloudStorageClass.getStorageClass(storageClass)).build();
}

The methods I am using to generate the file crc32c and md5 hashes using Google Guava Hashing APIs are listed below:

    private byte[] generateCrc32CheckSum(Path file) {
        byte[] crc32CheckSum = new byte[0];

        if(file != null) {
            byte[] buffer = new byte[SysConstants.FILE_READ_BUFFER_SIZE];
            int limit = -1;
            HashFunction crc32cHashFunc = Hashing.crc32c();
            Hasher crc32cHasher = crc32cHashFunc.newHasher();

            try (FileInputStream fis = new FileInputStream(file.toFile())) {
                while ((limit = fis.read(buffer)) > 0) {
                    crc32cHasher.putBytes(buffer, 0, limit);
                }

//                    crc32CheckSum = crc32cHasher.hash().asBytes();
                crc32CheckSum = Ints.toByteArray(crc32cHasher.hash().asInt());

            } catch (IOException e) {
                e.printStackTrace(); 
            }
        }

        return crc32CheckSum;
    }

    private byte[] generateMd5Hash(Path file) {
        byte[] md5Hash = new byte[0];

        if(file != null) {
            byte[] buffer = new byte[SysConstants.FILE_READ_BUFFER_SIZE];
            int limit = -1;
            HashFunction md5HashFunc = Hashing.md5();
            Hasher md5Hasher = md5HashFunc.newHasher();

            try (FileInputStream fis = new FileInputStream(file.toFile())) {
                while ((limit = fis.read(buffer)) > 0) {
                    md5Hasher.putBytes(buffer, 0, limit);
                }

                md5Hash = md5Hasher.hash().asBytes();

            } catch (IOException e) {
                e.printStackTrace();            
            }
        }

        return md5Hash;
    }

I then do the Base64 encoding in the following method:

public void queueFileWithSizeAndIntegrityCheckToken(SourceEvent sourceEvent, FileTransferStatus queuedStatus) {
    if(isFileTransferReady(sourceEvent)) {
        String fileName = sourceEvent.getFileName();
        Path filePath = Paths.get(fileName);

        if(filePath != null) {
            long fileSize = filePath.toFile().length();

            if(fileSize > 0) {
                sourceEvent.setFileSize(fileSize);

                if (isIntegrityCheckEnabled) {
                    if (integrityCheckType.equalsIgnoreCase(FileIntegrityCheckType.CRC32C.getIntegrityCheckTokenType())) {
                        byte[] crc32CheckSum = generateCrc32CheckSum(filePath);

                        if(crc32CheckSum.length > 0) {
                            sourceEvent.setFileHash(BaseEncoding.base64().encode(crc32CheckSum));
                            sourceEvent.setFileHashType(FileIntegrityCheckType.CRC32C.getCheckType());
                        }
                    } else if (integrityCheckType.equalsIgnoreCase(FileIntegrityCheckType.MD5.getIntegrityCheckTokenType())) {
                        byte[] md5Token = generateMd5Hash(filePath);

                        if(md5Token.length > 0) {
                            sourceEvent.setFileHash(BaseEncoding.base64().encode(md5Token));
                            sourceEvent.setFileHashType(FileIntegrityCheckType.MD5.getCheckType());
                        }
                    }
                }

                fileTransferLogService.updateSourceEventStatus(sourceEvent,
                        queuedStatus);
            }
        }
    }
}

Can someone kindly review and let me know what I am doing wrong? Thank you.

Answer 1

The above code works and there is no issue with the Google Cloud Storage service or the Google Cloud Storage Java client libraries.

The issue has been resolved and seems to have been caused by transaction settings in Spring which apparently affected proper processing of the files to be uploaded and the generation of the required crc32c and md5 hash values.