CODEWITHSUNDEEP
javatomcatrmijmx
sorin
sorin

Reputation: 170798

Java JMX refusing connections with really weird case

I do have consistent problems enabling JMX on some servers. It seems that while this works on some, some of them are rejecting connection in a strange way.

The port open and I can telnet to it from both localhost or other hosts.

Here is what I get from jmxterm when I try to connect to one of the faulty ones.

This happens if the port is opened:

$>open 10.80.16.195:8091
#RuntimeIOException: Runtime IO exception: Connection refused to host: 127.0.0.1; nested exception is:
    java.net.ConnectException: Connection refused

THis happens if the port is really closed:

$>open 10.80.16.195:9999
#RuntimeIOException: Runtime IO exception: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 10.80.16.195; nested exception is:
    java.net.ConnectException: Connection refused]

Here are the parameters used to connect to start the JVM (tomcat): /usr/lib/jvm/java-7-oracle/bin/java -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8091 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=10.80.16.195 -classpath /opt/Confluence.5/bin/bootstrap.jar -Dcatalina.base=/opt/Confluence.5 -Dcatalina.home=/opt/Confluence.5 org.apache.catalina.startup.Bootstrap start

java -version
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)

I observed similar complains from others but nobody provided a clear cause for this. It's less likely to be the JMX client, as I used 3-4 ones and all seem to have similar behaviors, working with some servers and failing on others.

I do know that /etc/hosts may be linked to this because, i remember seeing the 127.0.1.1 reported in some cases and I do know that Debian machines do have this setup. 

127.0.0.1 localhost
127.0.0.1 dowa-01.my.domain.com dowa-01
# by default the previous line had 127.0.1.1 instead and this was causing problems with JMX    

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

As you probably observed the JMX is binded to the routable address of the machine, that's because that's the only way to allow connections from outside, if you manually specify the IP.

Now what's the problem?

Upvotes: 1

Views: 5086

Answers (2)

sorin
sorin

Reputation: 170798

The cause of this problem is that you have to manually specify the public IP of the machine, otherwise it JMX/RMI will fail to bind correctly.

IP=$(ifconfig | sed -En 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p')
# if we have several IPs found, we pick the first one
for IP in $IP
do
  break
  done
java .... -Djava.rmi.server.hostname=${IP}

Upvotes: 1

neildo
neildo

Reputation: 2386

In /etc/hosts on the server, I would replace this line:

127.0.0.1 dowa-01.my.domain.com dowa-01

With this:

10.80.16.195 dowa-01.my.domain.com dowa-01

The JVM on the server will try to resolve the the IP address for localhost and get 127.0.0.1. My guess is that JMX is then only accepting remote connections on 127.0.0.1 which would cause problems. Instead you want Java to recognize the real IP and listen on that.

Upvotes: 3

Related Questions