CODEWITHSUNDEEP
javaspringamazon-web-servicesamazon-s3
Luis Miguel
Luis Miguel

Reputation: 605

Best approach to use TransferManager to upload big file to an Amazon S3 bucket

Currently I'm trying to upload a file to an amazon S3 bucket, I've been doing some research on that and found that the class TransferManager will split the file into small chunks to upload it parallelized using multiple Threads if the file is big enough. Right now in the application, we're creating one single instance of AmazonS3 Client (one single bean created at the start of the application) and using that AmazonS3 Client to create an instance of the TransferManager class for each file the user needs to uploads a file, after the upload of the file is completed (this is checked by waitForCompletion method of the TransferManager) we're calling the method transferManager.shutdownNow(false) to close all the threads created by it, something like this:

@AutoWired
private AmazonS3 s3Client;


   /**
 * Uploads a file using TransferManager from a MultipartFile.
 */
public String uploadFileParallelized(MultipartFile file) {
  String fileName;
  TransferManager transferManager =
      TransferManagerBuilder.standard().withS3Client(this.s3Client).build();

  try {
    String extension = getExtensionFromBytes(file.getBytes());
    fileName = getFileName(extension);

    ObjectMetadata metadata = new ObjectMetadata();
    metadata.setContentLength(file.getSize());
    metadata.setContentType(file.getContentType());

    Upload upload = transferManager.upload(getBucketName(), this.folder + fileName,
        file.getInputStream(), metadata);
    upload.waitForCompletion();
  } catch (IOException | AmazonClientException | InterruptedException e) {
    throw new FileUploadException("Couldn't upload the file to S3: " + e.getLocalizedMessage(),
        e);
  } finally {
    transferManager.shutdownNow(false);
  }


  return fileName;
}

The doubts i still have:

  1. I'm not sure whether you should instantiate a transferManager for each upload or should we use only one instance of the TransferManager (maybe a bean) but in that case I won't be able to call the transferManager.shutdownNow(false) method cause I won't be able to use it for the second upload.

  2. Does the TransferManager close all the threads used to upload the file even if I don't call the shutdownNow method?

  3. Is it ok to use the same AmazonS3 Client to create multiple instances of the TransferManager or should we create an S3Client for each TransferManager that we need?

  4. Should we use TransferManager even if the file is a small one (let's say its smaller than 5 MB)

Upvotes: 5

Views: 12325

Answers (1)

gruuuvy
gruuuvy

Reputation: 2129

https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/transfer/TransferManager.html

  1. You should call shutdownNow on your single TransferManager instance after all your uploads are done.

    TransferManager is responsible for managing resources such as connections and threads; share a single instance of TransferManager whenever possible. TransferManager, like all the client classes in the AWS SDK for Java, is thread safe. Call TransferManager.shutdownNow() to release the resources once the transfer is complete.

  2. Yes.

    By default, the thread pool will shutdown when the transfer manager instance is garbage collected.

  3. You should use the same S3 client. It is thread safe.

  4. The minUploadPartSize is 5MB and the minUploadThresholdSize is 16MB. You can use the TransferManager to upload for all your files, and it will break down the files depending on the file size. These options are configurable. If you know all your files are relatively small, it might be easier to use putObject instead.

Upvotes: 4

Related Questions