Using HTTPBuilder and getting "java.net.SocketException: No buffer space available (maximum connections reached?): connect"

Question

I'm using HTTPBuilder to build a client for the REST API of a NoSQL database. It works fine except when processing a large data set. After a large number of calls the client throws this exception:

java.net.SocketException: No buffer space available (maximum connections reached?): connect
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
at  org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:640)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:446)
at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:417)
at groovyx.net.http.HTTPBuilder.request(HTTPBuilder.java:366)
at groovyx.net.http.HTTPBuilder$request.call(Unknown Source)
at scripts.groovy.prototypes.NeoClient.getRelationship(NeoClient.groovy:70)
at com.gsgenetics.genie.trait.Node$Edge.getTargetNode(Node.java:108)
at com.gsgenetics.genie.trait.Node$Edge.compareTo(Node.java:145)
at com.gsgenetics.genie.trait.Node$Edge.compareTo(Node.java:67)
at com.gsgenetics.genie.trait.Node.getEdges(Node.java:32)
at com.gsgenetics.genie.trait.Node.iterator(Node.java:384)
at com.gsgenetics.genie.GenomeStructuredGraph.add(GenomeStructuredGraph.java:131)
at com.gsgenetics.genie.GenomeStructuredGraph.add(GenomeStructuredGraph.java:197)
at com.gsgenetics.genie.GenomeStructuredGraph$add.call(Unknown Source)
at scripts.groovy.prototypes.read-ccds$_run_closure3.doCall(read-ccds.groovy:53)
at scripts.groovy.prototypes.read-ccds.run(read-ccds.groovy:50)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)

It looks like some resources are not being released but I can' figure out what exactly and how to get them to release. I am running this on Windows 7 but I get a similar error when running it on Linux.

Here's the code where I am instantiating and using HTTPBuilder:

import groovyx.net.http.*
import net.sf.json.JSONArray
import net.sf.json.JSONObject

import static groovyx.net.http.ContentType.*
import static groovyx.net.http.Method.*

class NeoClient {

    static final URL = 'http://some IP address'
//    static def http = new HTTPBuilder( URL )
    static net.sf.json.JSONObject query(statement, params=null, success=null, error=null) {
        def http = new HTTPBuilder( URL )
        net.sf.json.JSONObject returnJson = []
        http.request( POST, JSON ) {
            uri.path = '/db/data/cypher/'
            headers.'X-Stream' = 'true'
            requestContentType = JSON
            body =  [ query : statement , params : params ?: [:] ]


            response.success = { resp, json ->
                if (success) success(json)
                else {
                    println "Status ${resp.statusLine} Columns ${json.columns}\nData: ${json.data}"
                }
                returnJson = json
            }

            response.failure = { resp, message ->
                def result=[status:resp.statusLine.statusCode,statusText:resp.statusLine.reasonPhrase]
                result.headers = resp.headers.collect { h -> [ (h.name) : h.value ] }
                result.message = message
                if (error) {
                    error(result)
                } else {
                    println "Status: ${result.status} : ${result.statusText} "
                    println "Headers: ${result.headers}"
                    println "Message: ${result.message}"
                }
                throw new Exception("Neo Client Exception");
            }
        }
    }
}

Note that I tried having a single instance of HTTPBuilder in my static class NeoClient, and also creating a new instance of HTTPBuilder for each call to "query", and I get the same exception in either case.

Any ideas would be greatly appreciated.

Thanks

Answer 1

You should use the HTTPBuilder.shutdown method for releasing any system resources held by this instance.

In addition, creating a new HTTPBuilder for each request is not the best approach in terms of performance and resource management. A better approach can be having a single instance of HttpBuilder which handles multiple requests. To do so, you will need to override the default behavior of HttpBuilder to use a thread safe connection manager:

private static class CustomHttpBuilder extends HttpBuilder {
    protected AbstractHttpClient createClient( HttpParams params ) {
       def connManager = new MultiThreadedHttpConnectionManager()
       def connManagerParams = new HttpConnectionManagerParams()
       connManagerParams.maxTotalConnections = 50 // default is 20
       connManagerParams.defaultMaxConnectionsPerHost = 50 // default is 2
       connManager.params = connManagerParams
       new HttpClient(connManager)
   }
} 

I got this code example from the Grails users forum. This will allow you to use a single instance of the CustomHttpBuilder for multiple requests.