CODEWITHSUNDEEP
javaamazon-s3
daydreamer
daydreamer

Reputation: 92149

Process dies when trying to put InputStream to Amazon S3

This is what I do to write to InputStream

public OutputStream getOutputStream(@Nonnull final String uniqueId) throws PersistenceException {
        final PipedOutputStream outputStream = new PipedOutputStream();
        final PipedInputStream inputStream;
        try {
            inputStream = new PipedInputStream(outputStream);
            new Thread(
                    new Runnable() {
                        @Override
                        public void run() {
                            PutObjectRequest putObjectRequest = new PutObjectRequest("haritdev.sunrun", "sample.file.key", inputStream, new ObjectMetadata());
                            PutObjectResult result = amazonS3Client.putObject(putObjectRequest);
                            LOGGER.info("result - " + result.toString());
                            try {
                                inputStream.close();
                            } catch (IOException e) {

                            }
                        }
                    }
            ).start();
        } catch (AmazonS3Exception e) {
            throw new PersistenceException("could not generate output stream for " + uniqueId, e);
        } catch (IOException e) {
            throw new PersistenceException("could not generate input stream for S3 for " + uniqueId, e);
        }
         try {
            return new GZIPOutputStream(outputStream);
        } catch (IOException e) {
            LOGGER.error(e.getMessage(), e);
            throw new PersistenceException("Failed to get output stream for " + uniqueId + ": " + e.getMessage(), e);
        }
    }

and in the following method, I see my process die 

protected <X extends AmazonWebServiceRequest> Request<X> createRequest(String bucketName, String key, X originalRequest, HttpMethodName httpMethod) {
    Request<X> request = new DefaultRequest<X>(originalRequest, Constants.S3_SERVICE_NAME);
    request.setHttpMethod(httpMethod);
    if (bucketNameUtils.isDNSBucketName(bucketName)) {
        request.setEndpoint(convertToVirtualHostEndpoint(bucketName));
        request.setResourcePath(ServiceUtils.urlEncode(key));
    } else {
        request.setEndpoint(endpoint);

        if (bucketName != null) {
            /*
             * We don't URL encode the bucket name, since it shouldn't
             * contain any characters that need to be encoded based on
             * Amazon S3's naming restrictions.
             */
            request.setResourcePath(bucketName + "/"
                    + (key != null ? ServiceUtils.urlEncode(key) : ""));
        }
    }

    return request;
}

The process fails on request.setResourcePath(ServiceUtils.urlEncode(key)); and I can't even debug because of that, even though the key is valid name and is not NULL

Can someone please help?

This is how the request looks before dying

request = {com.amazonaws.DefaultRequest@1931}"PUT https://my.bucket.s3.amazonaws.com / "
resourcePath = null
parameters = {java.util.HashMap@1959} size = 0
headers = {java.util.HashMap@1963} size = 0
endpoint = {java.net.URI@1965}"https://my.bucket.s3.amazonaws.com"
serviceName = {java.lang.String@1910}"Amazon S3"
originalRequest = {com.amazonaws.services.s3.model.PutObjectRequest@1285}
httpMethod = {com.amazonaws.http.HttpMethodName@1286}"PUT"
content = null

Upvotes: 3

Views: 2089

Answers (2)

user2047665
user2047665

Reputation: 81

Here is what I tried and worked - 

  try (PipedOutputStream pipedOutputStream = new PipedOutputStream();
     PipedInputStream pipedInputStream = new PipedInputStream()) {
            new Thread(new Runnable() {

            public void run() {
                try {
                      // write some data to pipedOutputStream
                } catch (IOException e) {
                   // handle exception
                }
            }
            }).start();
       PutObjectRequest putObjectRequest = new PutObjectRequest(BUCKET, FILE_NAME, pipedInputStream, new ObjectMetadata());
       s3Client.putObject(putObjectRequest);
}

This code worked with S3 throwing warning that content-length is not set and s3 will be buffered and could result in OutOfMemoryException. I am not convinced about any cheap method to set content-length in ObjectMetadata just to get rid of this message and hopefully AWS SDK would not be stream the whole stream into memory just to find the content-length.

Upvotes: 1

Tinclon
Tinclon

Reputation: 967

I tried the same approach and it failed for me as well.

I ended up writing all my data to the output stream first, and then initiating the upload to S3 after copying the data from the output stream to the input stream:

...
// Data written to outputStream here
...
byte[] byteArray = outputStream.toByteArray();
amazonS3Client.uploadPart(new UploadPartRequest()
  .withBucketName(bucket)
  .withKey(key)
  .withInputStream(new ByteArrayInputStream(byteArray))
  .withPartSize(byteArray.length)
  .withUploadId(uploadId)
  .withPartNumber(partNumber));

Kind of defeats the purpose of writing to a stream, if the entire data block has to be written and copied in memory before the upload to S3 can even begin, but it's the only way I could get it to work.

Upvotes: 1

Related Questions