CODEWITHSUNDEEP
javamultithreadingamazon-s3
litong
litong

Reputation: 11

Using multithreading to upload pictures to AmasonS3 as a storage platform

There are only single uploads and partial uploads in the official API, and the thread pool（20） uploads I write have a connection timeout error:

Exception in thread "pool-1-thread-12" com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to s3.amazonaws.com:80 [s3.amazonaws.com/54.231.66.16] failed: connect timed out

I saw API of AmasonS3, but I didn't find the answer.

import com.amazonaws.ClientConfiguration;
import com.amazonaws.Protocol;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.gaodig.stream.common.Constants;
import com.gaodig.stream.config.VEEConfig;

public class CephS3Client {

    private static volatile AmazonS3 s3Client = null;

    public static AmazonS3 getClient() {
        if (s3Client == null) {
            synchronized (CephS3Client.class) {
                if (s3Client == null) {
                    AWSCredentials credentials = new BasicAWSCredentials(Constants.CephS3AccessKey,
                            Constants.CephS3SecretKey);
                    ClientConfiguration clientConfig = new ClientConfiguration();
                    clientConfig.setProtocol(Protocol.HTTP);
                    clientConfig.setSignerOverride("S3SignerType");

                    // AmazonS3 conn = new AmazonS3Client(credentials,
                    // clientConfig);
                    s3Client = new AmazonS3Client(credentials, clientConfig);

                    s3Client.setEndpoint(VEEConfig.CEPH_ENDPOINT());
                }
            }
        }
        return s3Client;
    }

}

Upvotes: 1

Views: 701

Answers (1)

Egor K.
Egor K.

Reputation: 171

If I understand your problem right, this feature was not implemented in java sdk 1.0. But in new amazon java sdk they implement a lot of async features to dynamo db and s3, also for http client. So now You can use async client for s3. Example: 

public static void main(String[] args) {
S3AsyncClient client = S3AsyncClient.create();
CompletableFuture<PutObjectResponse> future = client.putObject(
        PutObjectRequest.builder()
                        .bucket(BUCKET)
                        .key(KEY)
                        .build(),
        AsyncRequestProvider.fromFile(Paths.get("myfile.in"))
);
future.whenComplete((resp, err) -> {
    try {
        if (resp != null) {
            System.out.println(resp);
        } else {
            // Handle error
            err.printStackTrace();
        }
    } finally {
        // Lets the application shut down. Only close the client when you are completely done with it.
        FunctionalUtils.invokeSafely(client::close);
    }
});

}

you can read this docs new AWS SDK for Java maybe that helps you

Upvotes: 1

Related Questions