CODEWITHSUNDEEP
javaspring-bootamazon-s3aws-sdk
Christian Triebstein
Christian Triebstein

Reputation: 425

Amazon AWS Client Timeout with lots of requests

We have a Spring Boot application that stores multimedia files (up to 100 MB in size) in a S3 compatible cloud storage. The application receives these files via REST call or an AMQP message broker (RabbitMQ).

Usually the load on the system is moderate so that there is no problem at all. However we encounter problems with accessing the S3 when there is heavy load on the system. Currently we are working around this issue with using a pool of 10 AmazonS3Clients that are assigned randomly to the calling process. This actually improves the issue but does not fix the problem. When the load is too high (meaning plenty of write and read operations) we encounter an exception of this sort:

 com.amazonaws.AmazonClientException: Unable to execute HTTP request: connect timed out
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:299)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:170)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2648)
    at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1049)
    at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:924)

We're using the 1.3.8 version of the aws-java-sdk and cannot easily update to a newer version due to the region settings in the newer versions. The signing algorithm prevents us from accessing our buckets properly in the newest version.

The implementation looks as follows:

Initialization (at constructor level):

ClientConfiguration clientConfiguration = new ClientConfiguration();
clientConfiguration.setConnectionTimeout(AWS_CONNECTION_TIMEOUT);
clientConfiguration.setMaxConnections(AWS_MAX_CONNECTIONS);
AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);

for (int i = 0; i < AWS_MAX_CLIENTS; i++) {
     s3[i] = new AmazonS3Client(credentials, clientConfiguration);
     s3[i].setEndpoint(endpoint);       
}

Put:

int i = getRandomClient();
s3[i].putObject(bucketName, key, file);

Get:

ReadableByteChannel channel;
try {
    int i = getRandomClient();
    S3Object object = s3[i].getObject(bucketName, addPrefix(fileId, prefix));
    S3ObjectInputStream stream = object.getObjectContent();
    channel = Channels.newChannel(stream);

    File file = File.createTempFile(fileId, "");
    try (WritableByteChannel outChannel = Channels.newChannel(new FileOutputStream(file))) {
        ByteBuffer buffer = ByteBuffer.allocate(8192);
        int read;
        while ((read = channel.read(buffer)) > 0) {
            buffer.rewind();
            buffer.limit(read);
            while (read > 0) {
                read -= outChannel.write(buffer);
            }
            buffer.clear();
        }
        IOUtils.closeQuietly(stream);
        return file;
    }        
}
catch (AmazonClientException e) {
    if (!isMissingKey(e)) {
        throw new IOException(e);
    }
}
finally {
    if (channel != null) {
        channel.close();
    }
}

It is pretty clear that the limited number of connections and clients is the bottleneck. There are plenty of ways how we could tweak the implementation to work properly. We could of course limit the number of consumers listening to the message broker. We could also increase the timeouts, number and connections of aws clients or limit the throughput in the service layer. However we're looking for a more sophisticated approach to handle things here.

Is there any way to tell whether or not a designated client can currently be used or has too many open connections? Is there any way one could let the client wait for the next free connection?

Upvotes: 6

Views: 11205

Answers (2)

Zach Johnson
Zach Johnson

Reputation: 96

If you are doing this due to a large number of HTTP errors, mostly being timeouts, it could also be that you need to close your S3Objects. If you don't close them, they become a resource hog and will cause these kinds of errors when sending requests to the S3 bucket.

https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/S3Object.html#close--

Upvotes: 2

thedarklord47
thedarklord47

Reputation: 3292

Increasing the number of clients is no different than increasing the connection pool size of a single client, except now you have to worry about pseudo-"load balancing" your array of clients with getRandomClient(). Additionally, there is significant overhead to creating multiple clients and maintaining an unnecessary number of connection pools. You are trying to reinvent the wheel.

One thing you can do is catch the Exception thrown during timeouts like so:

try {
    ... do s3 read/write ...
} catch (AmazonClientException ace) {
    if (ace.getCause() instanceof org.apache.http.conn.ConnectionPoolTimeoutException) {
        log.error("S3 connection pool timeout!");
    }
}

Use this to help tune your connection pool size. Basically just keep making it bigger until this is no longer your bottleneck.

Upvotes: 2

Related Questions