CODEWITHSUNDEEP
cassandrathrift
beterthanlife
beterthanlife

Reputation: 1668

Cassandra nodetool connection timed out

Im trying to use nodetool to check the status of my cluster, but its unable to connect.

My cassandra.yaml is configured with listen_address and rpc_address set as the server IP (e.g. 10.10.10.266).

Im able to connect through cqlsh and cassandra-cli using the same IP, but when I connect to nodetool it doesnt work.

/bin$ nodetool -h 10.10.10.266 ring
Failed to connect to '10.10.10.266:7199': Connection has timed out

I dont think I have a firewall enabled on the server (Ubuntu). Im running this directly on the server in question, so I wouldnt have thought it would be a firewall issue anyway.

Upvotes: 4

Views: 13870

Answers (5)

Kashyap
Kashyap

Reputation: 17546

As someone mentioned, it connects to the JMX port.

You can find the JMX port:

  • In /etc/cassandra/cassandra-env.sh. This won't work for ccm based local clusters OR
  • (my fav) by looking at the command-line of Cassandra node process running on the node.

My case was a cluster created locally using ccm so all my nodes were running on same host with different JMX port.

vagrant@triforce:~$ ps -eaf | grep cassandra | grepi -o " [^ ]*jmx.local.port[^ ]* "
 -Dcassandra.jmx.local.port=7100
 -Dcassandra.jmx.local.port=7300
 -Dcassandra.jmx.local.port=7200
vagrant@triforce:~$

This is because I have 3 nodes running on the localhost.

vagrant@triforce:~$ nodetool -p 7100 ring

Datacenter: datacenter1
==========
Address    Rack        Status State   Load            Owns                Token
                                                                          3074457345618258602
127.0.0.1  rack1       Up     Normal  64.65 MB        33.33%              -9223372036854775808
127.0.0.2  rack1       Up     Normal  65.26 MB        33.33%              -3074457345618258603
127.0.0.3  rack1       Up     Normal  65.92 MB        33.33%              3074457345618258602

vagrant@triforce:~$

Upvotes: 0

Jacek L.
Jacek L.

Reputation: 1416

You probably need to uncomment the following parameter in cassandra-env.sh:

-Djava.rmi.server.hostname=<public name>

Replace with the address of the interface you want the jmx interface to listen on.

Upvotes: 7

Shawn Parrish
Shawn Parrish

Reputation: 101

I also saw this same issue but it turned out to be some weirdness in my hosts file that was preventing JMX from binding to the interfaces.

Specifically, the host file had an entry for the external IP address with the hostname. Our servers had two interfaces, one external and one for an internal network. Removing that hosts entry did the trick.

Upvotes: 1

joy
joy

Reputation: 3707

I was facing the same timeout issue. However I found that my cluster was not getting started properly because of token issue and I was getting "Host ID collision between active endpoint". Once i deleted data directory and restarted cluster then nodetool started working fine.

Upvotes: 1

Wildfire
Wildfire

Reputation: 6418

nodetool connects through JMX interface. By default it's listening on port 7199 (other tools use RPC interface listening on port 9160 by default). Check JMX settings in cassandra-env.sh file. Most likely JMX server is listening on wrong interface (or probably loopback interface).

Default JMX configuration section (cassandra ver. 1.1.5) contains link to troubleshooting guide:

# jmx: metrics and administration interface
# 
# add this if you're having trouble connecting:
# JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=<public name>"
# 
# see 
# https://blogs.oracle.com/jmxetc/entry/troubleshooting_connection_problems_in_jconsole
# for more on configuring JMX through firewalls, etc. (Short version:
# get it working with no firewall first.)
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT" 
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false" 
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=false" 
JVM_OPTS="$JVM_OPTS $JVM_EXTRA_OPTS"

It also worths to list all network interfaces using ifconfig and try telnet'ing port 7199 on all interfaces.

Upvotes: 3

Related Questions