Adding cluster: “Error creating cluster: Call to /cluster-configs timed out.”

Question

(Continuing the discussion with the same title on the datastax forum). I was able to reproduce the issue where opscenter is unable to connect to a 2.0.1 cluster, using "Use existing cluster", failing with the message "Error creating cluster: Call to /cluster-configs timed out.". It is related to having "rpc_server_type: hsha" in cassandra.yaml.

I reproduced it as follows:

(1) Installed ubuntu 12.04 (x86-64 architecture) in qemu. Updated it to the latest version of all packages. Configured it with a static ip address (192.168.77.3). Qemu networking was set up so that the host machine and the qemu virtual machine can communicate.

(2) Downloaded Sun jre-7u45-linux-x64.tar.gz and installed it. Installed libjna-java. This all done as per the datastax installing on Debian/ubuntu docs.

(3) Installed datastax cassandra 2.0.1 using the Debian package, as described in the datastax docs.

(4) Made the following changes to cassandra.yaml:

seeds: "192.168.77.3" listen_address: 192.168.77.3 rpc_address: 192.168.77.3 rpc_server_type: hsha NB: To see the failure, it is essential to use hsha.

(5) Stopped the cassandra instance (Debian automatically starts it when installed). Note that the init script doesn't work for stopping cassandra (this is a new problem with cassandra 2.0), so I had to kill the process by hand. This is a trap: you may think you restarted cassandra and that it has taken your configuration changes into account, only it hasn't because you are still running the old instance.

(6) Cleared out instance data: sudo rm -fr /var/lib/cassandra/*

(7) Started a new cassandra instance. Checked that nodetool could connect to it from both the virtual machine (i.e. running locally) and from the host machine.

(8) Tried to add the cluster from opscenter-free running on the host (i.e. not running on the virtual machine). opscenter version 3.2.2, ubuntu 13.10. As no cluster had been added yet, I got the "Welcome to Datastax opscenter" dialog, with "Create New Cluster" or "Use Existing cluster". Chose "Use Existing Cluster". Added the ip address (192.168.77.3) of the qemu virtual machine instance. Clicked "Save cluster". This failed with "Error creating cluster: Call to /cluster-configs timed out".

The opscenter log:

2013-10-28 11:59:04+0100 []  INFO: Log opened.
2013-10-28 11:59:04+0100 []  INFO: twistd 10.2.0 (/usr/bin/python2.7 2.7.5) starting up.
2013-10-28 11:59:04+0100 []  INFO: reactor class: twisted.internet.selectreactor.SelectReactor.
2013-10-28 11:59:04+0100 []  INFO: set uid/gid 0/0
2013-10-28 11:59:04+0100 []  INFO: Logging level set to 'info'
2013-10-28 11:59:04+0100 []  INFO: OpsCenter version: 3.2.2
2013-10-28 11:59:04+0100 []  INFO: Compatible agent version: 3.2.2
2013-10-28 11:59:04+0100 []  INFO: No clusters are configured yet, checking to see if a config migration is needed
2013-10-28 11:59:04+0100 []  INFO: Main config does not appear to include a cluster configuration, skipping migration
2013-10-28 11:59:04+0100 []  INFO: No clusters are configured
2013-10-28 11:59:04+0100 []  INFO: HTTP BASIC authentication disabled
2013-10-28 11:59:04+0100 []  INFO: Starting webserver with ssl disabled.
2013-10-28 11:59:04+0100 []  INFO: SSL agent communication enabled
2013-10-28 11:59:04+0100 []  INFO: opscenterd.WebServer.OpsCenterdWebServer starting on 8888
2013-10-28 11:59:04+0100 []  INFO: Starting factory 
2013-10-28 11:59:04+0100 []  INFO: morbid.morbid.StompFactory starting on 61619
2013-10-28 11:59:04+0100 []  INFO: Starting factory 
2013-10-28 11:59:04+0100 []  INFO: Configuring agent communication with ssl support enabled.
2013-10-28 11:59:04+0100 []  INFO: morbid.morbid.StompFactory starting on 61620
2013-10-28 11:59:04+0100 []  INFO: OS Version: Linux version 3.11.0-12-generic (buildd@allspice) (gcc version 4.8.1 (Ubuntu/Linaro 4.8.1-10ubuntu7) ) #19-Ubuntu SMP Wed Oct 9 16:20:46 UTC 2013
2013-10-28 11:59:04+0100 []  INFO: CPU Info: ['2401.000', '1200.000', '1200.000', '2401.000', '1200.000', '1200.000', '1200.000', '2401.000']
2013-10-28 11:59:04+0100 []  INFO: Mem Info: 15979MB
2013-10-28 11:59:04+0100 []  INFO: Package Manager: Unknown
2013-10-28 12:03:02+0100 []  INFO: Starting factory 
2013-10-28 12:03:02+0100 []  INFO: Adding new cluster 'Test_Cluster': {u'jmx': {u'username': u'', u'password': u'', u'port': u'7199'}, 'kerberos_client_principals': {}, 'kerberos': {}, u'agents': {}, 'kerberos_hostnames': {}, 'kerberos_services': {}, u'cassandra': {u'username': u'', u'seed_hosts': u'192.168.77.3', u'api_port': u'9160', u'password': u''}}
2013-10-28 12:03:02+0100 []  INFO: Starting new cluster services for Test_Cluster
2013-10-28 12:03:02+0100 [Test_Cluster]  INFO: Starting services for cluster Test_Cluster
2013-10-28 12:03:02+0100 []  INFO: Metric caching enabled with 50 points and 1000 metrics cached
2013-10-28 12:03:02+0100 []  INFO: Starting PushService
2013-10-28 12:03:02+0100 [Test_Cluster]  INFO: Starting CassandraCluster service
2013-10-28 12:03:02+0100 [Test_Cluster]  INFO: agent_config items: {'cassandra_log_location': '/var/log/cassandra/system.log', 'thrift_port': 9160, 'thrift_ssl_truststore': None, 'rollups300_ttl': 2419200, 'rollups86400_ttl': -1, 'jmx_port': 7199, 'metrics_ignored_solr_cores': '', 'api_port': '61621', 'metrics_enabled': 1, 'thrift_ssl_truststore_type': 'JKS', 'kerberos_use_ticket_cache': True, 'kerberos_renew_tgt': True, 'rollups60_ttl': 604800, 'cassandra_install_location': '', 'rollups7200_ttl': 31536000, 'kerberos_debug': False, 'storage_keyspace': 'OpsCenter', 'ec2_metadata_api_host': '169.254.169.254', 'provisioning': 0, 'kerberos_use_keytab': True, 'metrics_ignored_column_families': '', 'thrift_ssl_truststore_password': None, 'metrics_ignored_keyspaces': 'system, system_traces, system_auth, dse_auth, OpsCenter'}
2013-10-28 12:03:02+0100 []  INFO: Stopping factory

Adding cluster: “Error creating cluster: Call to /cluster-configs timed out.”

Answers (1)

Related Questions