Java: Intermittent UnknownHostException

Question

In my application which runs on Jboss server, we make an http connection to another application. Both the application lives in the same datacenter. We use the VIP to connect to the application. This works fine in production except for a very small percentage when we get "java.net.UnknownHostException". The percentage is very low (~0.2%), but still considering the huge volume of requests we get, the actual number cannot be ignored.

The error is intermittent and does not follow a regular pattern neither in terms of time, nor frequency. Sometimes we get this 200 times in a day and sometimes we do not get it for 3-4 consecutive days. Sometimes we even get it when the expected traffic is low (1-3 AM at night).

AFAIK, this particular exception comes when the DNS is not able to resolve the IP for a given hostname. We looked into the DNS settings/configurations etc and everything looks good. Do not want to use the IP address directly in the properties file or in /etc/hosts file.

At this point, I am not sure what I can do next to debug further. Any help here/guidance would be much appreciated.

    Caused by: java.net.UnknownHostException: <VIP Name>
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:1.6.0_25]
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:850) ~[?:1.6.0_25]
at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1201) ~[?:1.6.0_25]
at java.net.InetAddress.getAllByName0(InetAddress.java:1154) ~[?:1.6.0_25]
at java.net.InetAddress.getAllByName(InetAddress.java:1084) ~[?:1.6.0_25]
at java.net.InetAddress.getAllByName(InetAddress.java:1020) ~[?:1.6.0_25]
at org.apache.http.impl.conn.DefaultClientConnectionOperator.resolveHostname(DefaultClientConnectionOperator.java:242) ~[httpclient-4.1.2.jar:4.1.2]
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:130) ~[httpclient-4.1.2.jar:4.1.2]
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:149) ~[httpclient-4.1.2.jar:4.1.2]
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121) ~[httpclient-4.1.2.jar:4.1.2]
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:573) ~[httpclient-4.1.2.jar:4.1.2]
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:425) ~[httpclient-4.1.2.jar:4.1.2]
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) ~[httpclient-4.1.2.jar:4.1.2]
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) ~[httpclient-4.1.2.jar:4.1.2]
at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:88) ~[spring-web-3.1.1.RELEASE.jar:3.1.1.RELEASE]
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:46) ~[spring-web-3.1.1.RELEASE.jar:3.1.1.RELEASE]
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:49) ~[spring-web-3.1.1.RELEASE.jar:3.1.1.RELEASE]
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:438) ~[spring-web-3.1.1.RELEASE.jar:3.1.1.RELEASE]
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:401) ~[spring-web-3.1.1.RELEASE.jar:3.1.1.RELEASE]