CODEWITHSUNDEEP
javaamazon-web-servicesamazon-s3
user2017502
user2017502

Reputation: 225

Java.net.SocketException: Connection reset when reading file from S3

I'm trying to read some files from S3 and do some processing on each file. I can get through some files, but I keep getting Java.net.SocketException: Connection reset at the same line in a particular file during processing. The file in question should be ok though because I can process it locally using the same class and method (conversionUtils.convert()).

Service class:

public class FileService {

  @Inject
  private S3Utils s3utils;

  private ConversionUtils conversionUtils = new ConversionUtils();

  public void processFile() {
    List<S3ObjectSummary> files = s3utils.getAllFiles();
    List<S3Object> unprocessedFiles = s3utils.getUnprocessedFiles(files);

    for(S3Object file: unprocessedFiles) {
      InputStream content = file.getObjectContent();
      List<Record> records = conversionUtils.convert(content); //Exception thrown here
    }
  }
}

S3Utils class: 

@Component
public class S3utils {
  @Inject AmazonS3 amazonS3;

  public List<S3ObjectSummary> getAllFiles() {
    ListObjectsV2Request request = new ListObjectsV2Request().withBucketName('something').withPrefix('some_prefix');
    ListObjectsV2Result result = amazonS3.listObjectsV2(request);;

    return result.getObjectSummaries();
  }

  public List<S3Object> getUnprocessedFiles(List<S3ObjectSummary> files) {
    //do some filtering here

    List<S3Object> unprocessedFiles = new ArrayList<>();
    for (S3ObjectSummary summary : filteredSummaries) {
      S3Object s3Object = amazonS3.getObject(new GetObjectRequest(summary.getBucketName(), summary.getKey()));
      unprocessedFiles.add(S3Object);
    }
    return unprocessedFiles;
  }
}

Config class:

@Configuration
public class Config {
  @Bean
  public AmazonS3 amazonS3() {
    return AmazonS3ClientBuilder.standard().withCredentials(new DefaultAWSCredentialsProviderChain()).build();
  }
}

I've read some other threads with similar errors where the issue was that the AmazonS3 client was being garbage collected, and therefore closing the stream, but I'm wondering if that's the case here. Any ideas on what the issue is? Thanks.

Upvotes: 7

Views: 11846

Answers (3)

Chris
Chris

Reputation: 5663

I was able to overcome this by setting the ClientConfig to keep the connection alive

AmazonS3ClientBuilder
   .standard()
   .withCredentials(new DefaultAWSCredentialsProviderChain())
   .withClientConfiguration(new ClientConfiguration().withTcpKeepAlive(true)) //Note withTcpKeepAlive method is on ClientConfiguration
   .build();

Upvotes: 4

Aswani
Aswani

Reputation: 79

After i upgrade s3 proxy to 1.6.0 and aws-java-sdk-s3 to 1.1.285 connection reset issue is fixed.

Upvotes: 0

jackk
jackk

Reputation: 1

I ran into same issue with aws sdk 1.X.

Strange thing here is, it's working fine in local machine and Connection reset while application runs on server.

Tried with withTcpKeepAlive(true), but result was same ie,

Java.net.SocketException: Connection reset

A workaround

I tried withRange(startPosition,endPosition) until I read all data and it worked.

Upvotes: 0

Related Questions