Reputation: 23137
I would like to upload a directory to S3 using the AWS Java SDK v2.
For example, how would I implement the following function?
fun uploadDirectory(bucket: String, prefix: String, directory: Path)
I would like the contents of directory
to be replicated at s3://bucket/prefix/
on S3.
The v2 SDK documentation has an example for uploading a single object, but there doesn't seem to be an equivalent to this Upload a Directory example from v1.
Upvotes: 1
Views: 2148
Reputation: 23137
You can implement it by using the following strategy:
Files.walk
to walk the directory, identifying all of the files.S3AsyncClient.putObject
.CompletableFuture.allOf
to combine all of the upload tasks, and wait for completion.This strategy uses the async client's default thread pool of 50 threads. This is working fine for me with directories that contain thousands of files.
The s3Prefix
here is the prefix to add to each object uploaded to the bucket, equivalent to the target directory.
fun uploadDirectory(s3Bucket: String, s3Prefix: String, directory: Path) {
require(directory.isDirectory())
Files.walk(directory).use { stream ->
stream.asSequence()
.filter { it.isRegularFile() }
.map { path ->
putObject(
s3Bucket = s3Bucket,
s3Key = "$s3Prefix/${directory.relativize(path)}",
path = path
)
}
.toList().toTypedArray()
}.let { CompletableFuture.allOf(*it) }.join()
}
private fun putObject(s3Bucket: String, s3Key: String, path: Path)
: CompletableFuture<PutObjectResponse> {
val request = PutObjectRequest.builder()
.bucket(s3Bucket)
.key(s3Key)
.build()
return s3AsyncClient.putObject(request, path)
}
Upvotes: 1
Reputation: 8355
TransferManager
along with some other high level libraries is not yet available in v2. so you will have to use v1 instead, from the migration guide
High-level libraries, such as the Amazon S3 Transfer Manager and the Amazon SQS Client-side Buffering, are not yet available in version 2.x. See the AWS SDK for Java 2.x changelog for a complete list of libraries.
If your application depends on these libraries, see Using both SDKs side-by-side to learn how to configure your pom.xml to use both 1.x and 2.x. Refer to the AWS SDK for Java 2.x changelog for updates about these libraries.
Upvotes: 0