Leon Csergity
Leon Csergity

Reputation: 311

AWS sdk 2 does not return from ".join()" that is called on the completable future on PutObject when uploading large files

We upgraded from version 1 to the version 2 sdk because support was cut for the first version of the SDK.

The old client sync client was switched out for the new async client and new transfer manager. Everything was kept as it was because the new code was called with the ".join()" on the completable futures. Everything was working fine until we realised that large files dont get uploaded.

The whole service is a spring boot service that handles file uploads and download to s3. The http-nio request thread logs the the "Before request" and does not finish afterwards.

I've logged everything I could by setting logging to DEBUG and TRACE and it seems that transfer manager finishes the multipart file upload because i see the log

2025-03-02T15:48:35,422 DEBUG [sdk-async-response-0-6] software.amazon.awssdk.services.s3.internal.multipart.GenericMultipartHelper : Sending completeMultipartUploadRequest, uploadId: ...xxxx

But the request thread does not continue. The transfer manager does not complete the completable future. This is how the method looks like:

public void putObject(PutObjectRequest putObjectRequest, InputStream inputStream) {
    ExecutorService executorService = null;
    try {
        executorService = Executors.newSingleThreadExecutor();
        transferManager.upload(UploadRequest.builder()
                        .putObjectRequest(putObjectRequest)
                        .requestBody(
                                AsyncRequestBody.fromInputStream(
                                        AsyncRequestBodyFromInputStreamConfiguration.builder()
                                                .executor(executorService)
                                                .contentLength(putObjectRequest.contentLength())
                                                .inputStream(inputStream)
                                                .build()
                                )
                        )
                        .build()
                )
                .completionFuture()
                .join();
    } catch (Exception ex) {
        throwErrorOnUploadFail(ex);
    } finally {
        if (Objects.nonNull(executorService)) {
            executorService.shutdown();
        }
    }
}

When debugging the code doesn't continue after the ".join()" and the initial request thread is held indefinitely

This same behaviour is observed on every request that is doing something with a large file (5 gigs plus) that returns a Completable future and has join called on it. The operation does not return from join and the request thread is held until the service is killed or stopped. But works as expected with smaller files.

I don't have much experience working with completable future and I feel like im missing something vital. Sorry for the wall of text btw.

Upvotes: -1

Views: 25

Answers (0)

Related Questions