cassandra 1.2 nodetool getting 'Failed to connect' when trying to connect to remote node

Question

I am running a 6 node cluster of cassandra 1.2 on an Amazon Web Service VPC with Oracle's 64-bit JVM version 1.7.0_10.

When I'm logged on to one of the nodes (ex. 10.0.12.200) I can run nodetool -h 10.0.12.200 status just fine.

However, if I try to use another ip address in the cluster (10.0.32.153) from that same terminal I get Failed to connect to '10.0.32.153:7199: Connection refused'.

On the 10.0.32.153 node I am trying to connect to I've made the following checks.

• From 10.0.12.200 I can run telnet 10.0.32.153 7199 and I get a connection, so it doesn't appear to be a security group/firewall issue to port 7199.

• On 10.0.32.153 if I run netstat -ant|grep 7199 I see

  tcp 0 0 0.0.0.0:7199 0.0.0.0:* LISTEN

  so cassandra does appear to be listening on the port

• The cassandra-env.sh file on 10.0.32.153 has all of the JVM_OPTS for jmx active

  -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false

The only shot in the dark I've seen while trying to solve this problem while searching the interwebs is to set the following:

JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=10.0.32.153"

But when I do this I don't even get a response. It just hangs.

Any guidance would be greatly appreciated.

Answer 1

The issue did end up being a firewall/security group issue. While it is true that the jmx port 7199 is used, apparently other ports are used randomly for rmi. Cassandra port usage - how are the ports used?

So the solution is to open up the firewalls then configure the cassandra-env.sh to include

JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=<ip>