CODEWITHSUNDEEP
amazon-s3
Shivam
Shivam

Reputation: 191

TransferManager vs PutRequestObject in AmazonS3

I am confused about Object single upload and multipart upload. Both are taking consumes same time. My code is below:

File file = new File("D:\\AmazonS3\\aws-java-sdk-1.8.3\\lib\\aws-java-sdk-1.8.3-javadoc.jar");
FileInputStream fis = new FileInputStream(file);

String keyName = System.currentTimeMillis()+"_aws-java-sdk-1.8.3-javadoc.jar";

ObjectMetadata metaData = new ObjectMetadata();
metaData.addUserMetadata("test","TEST");
//Object single upload
PutObjectRequest putobejcObjectRequest = new PutObjectRequest(BUCKET_NAME, keyName, fis,metaData);
putobejcObjectRequest.setMetadata(metaData);
s3client.putObject(putobejcObjectRequest);

//Object multipart upload
TransferManagerConfiguration configuration = new TransferManagerConfiguration();
    configuration.setMultipartUploadThreshold(5*com.amazonaws.services.s3.internal.Constants.MB);
TransferManager transferManager = new TransferManager(s3client);
transferManager.setConfiguration(configuration);

Upload upload = transferManager.upload(BUCKET_NAME, keyName, fis,metaData);
upload.waitForCompletion();
transferManager.shutdownNow();

Please help me, Is there any problem in my code.

Upvotes: 3

Views: 6425

Answers (2)

Alexandre G
Alexandre G

Reputation: 1693

When uploading options from a stream, callers must supply the size of options in the stream through the content length field in the ObjectMetadata parameter. If no content length is specified for the input stream, then TransferManager will attempt to buffer all the stream contents in memory and upload the options as a traditional, single part upload. Because the entire stream contents must be buffered in memory, this can be very expensive, and should be avoided whenever possible.

source: https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/transfer/TransferManager.html#upload-java.lang.String-java.lang.String-java.io.InputStream-com.amazonaws.services.s3.model.ObjectMetadata-

Upvotes: 1

zeldigas
zeldigas

Reputation: 641

I've faced the same problem and found (by inspecting SDK code) that transfer manager will use parallel parts upload only if you pass it file, not the InputStream.

Look at decision code from sdk (version 1.8.9):

if (TransferManagerUtils.isUploadParallelizable(putObjectRequest, isUsingEncryption)) {
   captureUploadStateIfPossible();
   uploadPartsInParallel(requestFactory, multipartUploadId);
   return null;
} else {
   return uploadPartsInSeries(requestFactory);
}

where is uploadParallelizable:

// Each uploaded part in an encrypted upload depends on the encryption context
// from the previous upload, so we cannot parallelize encrypted upload parts.
if (isUsingEncryption) return false;

// Otherwise, if there's a file, we can process the uploads concurrently.
return (getRequestFile(putObjectRequest) != null);

So, if you want to get advantage of parallel upload in parts, pass a file to TransferManager.

Upvotes: 3

Related Questions