CODEWITHSUNDEEP
kotlinamazon-s3file-uploadaws-java-sdk-2.xdirectory-upload
Adam Millerchip
Adam Millerchip

Reputation: 23137

AWS Java SDK v2: Upload a directory to S3

I would like to upload a directory to S3 using the AWS Java SDK v2.

For example, how would I implement the following function?

fun uploadDirectory(bucket: String, prefix: String, directory: Path)

I would like the contents of directory to be replicated at s3://bucket/prefix/ on S3.

The v2 SDK documentation has an example for uploading a single object, but there doesn't seem to be an equivalent to this Upload a Directory example from v1.

Upvotes: 1

Views: 2148

Answers (2)

Adam Millerchip
Adam Millerchip

Reputation: 23137

You can implement it by using the following strategy:

  1. Use Files.walk to walk the directory, identifying all of the files.
  2. Asynchronously upload the files using the SDK, via S3AsyncClient.putObject.
  3. Use CompletableFuture.allOf to combine all of the upload tasks, and wait for completion.

This strategy uses the async client's default thread pool of 50 threads. This is working fine for me with directories that contain thousands of files.

The s3Prefix here is the prefix to add to each object uploaded to the bucket, equivalent to the target directory.

fun uploadDirectory(s3Bucket: String, s3Prefix: String, directory: Path) {
    require(directory.isDirectory())

    Files.walk(directory).use { stream ->
        stream.asSequence()
            .filter { it.isRegularFile() }
            .map { path ->
                putObject(
                    s3Bucket = s3Bucket,
                    s3Key = "$s3Prefix/${directory.relativize(path)}",
                    path = path
                )
            }
            .toList().toTypedArray()
    }.let { CompletableFuture.allOf(*it) }.join()
}

private fun putObject(s3Bucket: String, s3Key: String, path: Path)
    : CompletableFuture<PutObjectResponse> {
    val request = PutObjectRequest.builder()
        .bucket(s3Bucket)
        .key(s3Key)
        .build()

    return s3AsyncClient.putObject(request, path)
}

Upvotes: 1

mightyWOZ
mightyWOZ

Reputation: 8355

TransferManager along with some other high level libraries is not yet available in v2. so you will have to use v1 instead, from the migration guide

High-level libraries, such as the Amazon S3 Transfer Manager and the Amazon SQS Client-side Buffering, are not yet available in version 2.x. See the AWS SDK for Java 2.x changelog for a complete list of libraries.

If your application depends on these libraries, see Using both SDKs side-by-side to learn how to configure your pom.xml to use both 1.x and 2.x. Refer to the AWS SDK for Java 2.x changelog for updates about these libraries.

Upvotes: 0

Related Questions