Multipart upload using AWS Java SDK hangs at 99%

Question

We are trying to upload a file of size 46GB to AWS S3. However the file upload hangs once it reaches 99%. We are using AWS java SDK for the mulitpart upload.

Following is the code which is used for multipart upload:

void startMultipartUpload(ObjectStoreAccess creds, List<Path> files) {
        List<File> filenames = files.stream().map(Path::toString).map(File::new).collect(Collectors.toList());
        TransferManager transferManager = transferManagerFactory.createTransferManager(creds);
        List<File> filesNotUploaded = new ArrayList<>();
        boolean isFileUploadSuccessful;
        Integer timeElapsed;
        for (File file : filenames) {
            isFileUploadSuccessful = false;
            timeElapsed = 0;
            try {
                String keyName = creds.getAwsS3TemporaryUploadCredentials().getKeyPrefix() + file.getName();
                PutObjectRequest request = new PutObjectRequest(creds.getAwsS3TemporaryUploadCredentials().getBucketName(), keyName, new File(file.getPath()));

                Upload upload = transferManager.upload(request);
                logger.info(String.format("Starting upload for : %s ", file.getName()));
                while (!upload.getState().equals(Transfer.TransferState.Completed)) {
                    Thread.sleep(1000);
                    timeElapsed++;
                    progressLogger.writeProgress(timeElapsed, upload.getProgress().getPercentTransferred());
                }
                upload.waitForCompletion();
                isFileUploadSuccessful = true;
                progressLogger.writeProgressStatus("...upload complete!\n");

            } catch (AmazonServiceException e) {
                String message = "AmazonServiceException: " + e.getMessage();
                logger.error(message);
            } catch (SdkClientException e) {
                String message = "SdkClientException: " + e.getMessage();
                logger.error(message);
            } catch (InterruptedException e) {
                String message = "InterruptedException: " + e.getMessage();
                logger.error(message);
                Thread.currentThread().interrupt();

            } finally {
                if (!isFileUploadSuccessful) {
                    String message = this.appMessages.getMessageByKey("FAIL_TO_UPLOAD_FILE") + " " + file.getPath();
                    logger.error(message);
                    filesNotUploaded.add(file);
                }

            }
        }

    }

Answer 1

Try using the AWS SDK for Java V2 and following this example that shows how to upload an object in parts. See:

https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/javav2/example_code/s3/src/main/java/com/example/s3/S3ObjectOperations.java