CODEWITHSUNDEEP
javaspring-bootamazon-s3wasabiwasabi-hot-cloud-storage
Space Cadet
Space Cadet

Reputation: 403

A NoSuchBucketException is being thrown when trying to copy files into a Wasabi bucket that already exists

I have a Spring Boot application that manages files stored in a Wasabi bucket. I have a requirement to copy files from one location in the bucket to another, so I am using an S3TransferManager to handle this operation. However, whenever I try to copy a file, the SDK reports that the bucket doesn't exist, even though I have visually verified its presence via the Wasabi console and also by checking for typos in the bucket name in my code.

Here is the configuration class that creates the beans I use to manage S3 operations

@Configuration
public class Config {
@Bean(name = "wasabiCredentialsProvider")
public AwsCredentialsProvider wasabiCredentialProvider(@Value("${the.accessKey}") String accessKey,
                                                     @Value("${the.secretKey}") String secretKey) {
    return StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey));
}



@Bean(name = "wasabiS3Async")
public S3AsyncClient wasabiS3Async(@Qualifier("wasabiCredentialsProvider") AwsCredentialsProvider awsCredentialsProvider,
                                   @Value("${the.region}") String region,
                                   @Value("${the.endpoint}") String endpoint) throws URISyntaxException {
    SdkAsyncHttpClient nettyHttpClient = NettyNioAsyncHttpClient.builder()
            .connectionAcquisitionTimeout(Duration.ofHours(3))
            .connectionTimeout(Duration.ofHours(3))
            .readTimeout(Duration.ofHours(3))
            .writeTimeout(Duration.ofHours(3))
            .maxConcurrency(100)
            .build();
    S3AsyncClientBuilder s3AsyncClientBuilder = S3AsyncClient.builder()
            .credentialsProvider(awsCredentialsProvider)
            .region(Region.of(region))
            .httpClient(nettyHttpClient);
    if (endpoint.contains("localhost")) {
        s3AsyncClientBuilder.endpointOverride(new URI(endpoint))
                .serviceConfiguration(S3Configuration.builder().pathStyleAccessEnabled(true).build());
    }
    return s3AsyncClientBuilder.build();
}

@Bean(name = "wasabiS3TransferManager")
public S3TransferManager wasabiS3TransferManager(final S3AsyncClient wasabiS3Async) {
    return S3TransferManager.builder()
            .s3Client(wasabiS3Async)
            .build();
}
}

Now here is the method where the problem occurs

private final S3TransferManager wasabiS3TransferManager;


public void copyFile(final String sourceFolder, final String targetFolder, final String sourceFileId, final String bucketName, final String targetFileId) {
    
    S3TransferManager s3TransferManager = wasabiS3TransferManager;
    String sourceBucketPath = String.format("%s/%s", sourceFolder, sourceFileId);
    String targetBucketPath = String.format("%s/%s", targetFolder, targetFileId);


    try {
        CopyObjectRequest copyObjectRequest = CopyObjectRequest.builder()
                .sourceBucket(bucketName)
                .sourceKey(sourceBucketPath)
                .destinationBucket(bucketName)
                .destinationKey(targetBucketPath)
                .build();
        CopyRequest copyRequest = CopyRequest.builder()
                .copyObjectRequest(copyObjectRequest)
                .build();
        Copy copy = s3TransferManager.copy(copyRequest);
        copy.completionFuture().join();
    } catch (Exception exception) {
        log.error("There was an error", exception.getMessage());
//Exception handling happens here 
    }
    
}

This is the exception message

software.amazon.awssdk.services.s3.model.NoSuchBucketException: The specified bucket does not exist (Service: S3, Status Code: 404, Request ID: <blah>, 
Extended Request ID:<blah>)

I had thought that this might be a problem with the region that's set on the S3AsyncClient bean but I'm injecting the correct value.

Any help is appreciated

Upvotes: 0

Views: 55

Answers (1)

Space Cadet
Space Cadet

Reputation: 403

The issue seems to be with the S3TransferManager class. If I just copy the file using an S3Client or S3AsyncClient then it copies successfully (I've not been able to visually confirm that the file has copied correctly because Web is not deployed in Dev2. But the operation completes without an exception).

What's strange is that the S3TransferManager bean that I am using takes an instance of the S3AsyncClient bean as its S3Client.

So I'm unsure why the S3AsyncClient works fine but the S3TransferManager, which uses the S3AsyncClient, doesn't.

Upvotes: 0

Related Questions