CODEWITHSUNDEEP
javaamazon-s3
Tim Harper
Tim Harper

Reputation: 2621

Java library AmazonS3Client.getObject never returns

I'm experiencing an issue in which it would appear that calls to AmazonS3Client.getObject never returns; this is happening to us every day, and I believe it's when S3 is running it's daily expiry job.

There doesn't seem to be a way to specify a timeout value for this request, and it's causing our thread pool to become blocked indefinitely, making our entire instance unresponsive when enough of them hang indefinitely.

Is there a way to overcome? I've gone down the Thread.interrupt path, but am not sure if it will that signal will abided.

Here is the stack trace for where it is getting stuck:

"image-s3-dispatcher-10187" - Thread t@10481
   java.lang.Thread.State: TIMED_WAITING
  at sun.misc.Unsafe.park(Native Method)
  - parking to wait for <7945c7e6> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  at java.util.concurrent.locks.LockSupport.parkUntil(LockSupport.java:267)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUntil(AbstractQueuedSynchronizer.java:2130)
  at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:129)
  at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:281)
  at org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:62)
  at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:176)
  at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:172)
  at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:100)
  at org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:212)
  at org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:199)
  at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
  at com.amazonaws.http.conn.$Proxy9.getConnection(Unknown Source)
  at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:456)
  at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
  at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
  at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:384)
  at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
  at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
  at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1111)
  at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:984)
  at com.spingo.s3client.S3Client$$anonfun$read$1.apply(S3Client.scala:110)

All of the 8 pending requests on which it is stuck are for files <=10kb, some of the requests are as old as 15 minutes.

Upvotes: 2

Views: 2528

Answers (1)

Tim Harper
Tim Harper

Reputation: 2621

After doing the following:

  • Upgrade to the latest aws-java-sdk (1.9.7) (from 1.9.6)
  • Explicitly closing the stream returned by S3Object#getObjectContent
  • Disabled the expiry rules on amazon.

... the problem, for the first time in a week, has stopped manifesting. Difficult to tell which, if any, of those things caused the improvement.

If it were due to not explicitly closing the streams, well, I've been using the aws-java-sdk that year for several months without any issue. And, it's odd how the problem manifested for all instances at the exact time each day.

If it's disabling the expiry rules on Amazon, well, I'll report what happens when I turn them back on.

If it's upgrading the latest aws-java-sdk, there was no mention of the observed infinite-block issue being addressed in the release notes.

While I'm eager to understand what was really the problem, I'm not eager enough that I am going to go out of my way to try and break it again. It's something that can only be reproduced at a specific time each day. :/

--

UPDATE

The issue actually wasn't fixed; it just happened to not occur one day. This morning it occurred again, at a different time.

Had more luck on the AWS forums. Seems as if it all traces back to an issue with the apache http library. Leaving bread-crumbs to benefit my fellow netizens:

https://forums.aws.amazon.com/thread.jspa?messageID=585774&#585774

--

UPDATE 2

Updating the apache http library didn't help. Nor did rolling back the AWS client.

There was an area in which we were checking for the existence of an object. To do so, we were calling AmazonS3Client#getObject and checking for an exception, thinking that S3Object#getObjectContent was the thing that actually fetched the stream. Reading the docs lead me to believe that. However, when reading the source code, I came to realize that getObjectContent is a simple getter and the library eager-allocates the stream. Since we didn't realize this fact, we didn't bother to call close on the stream if we didn't call S3Object#getObjectContent.

So, moral of the story: any time you call AmazonS3Client#getObject, close the stream.

Upvotes: 2

Related Questions