AuthenticationException when trying to connect to Cassandra with Java driver

Question

I'm trying to write a Java snippet that starts up Cassandra in a separate thread, and then connects using the driver and creates a few users. We had a script that used the Python based CQLSH client for this purpose, but ran into issues when we upgraded one of our servers to Ubuntu 16.04. Apparently there is a Python version incompatibility on Ubuntu (works on RHEL 7, but not Ubuntu 16.04). I even tried downgrading the Python version on Ubuntu, but still wasn't able to use the CQLSH client, so I've resorted to do this in Java. These are the Maven dependencies. Please note that I have tried with the default Guava version (16.0), but ran into some other issue, so I overrode it with 17.0.

[INFO] +- commons-io:commons-io:jar:1.3.2:compile
[INFO] +- org.apache.cassandra:cassandra-all:jar:2.2.4:compile
[INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.1.7:compile
[INFO] |  +- net.jpountz.lz4:lz4:jar:1.3.0:compile
[INFO] |  +- com.ning:compress-lzf:jar:0.8.4:compile
[INFO] |  +- commons-cli:commons-cli:jar:1.1:compile
[INFO] |  +- commons-codec:commons-codec:jar:1.9:compile
[INFO] |  +- org.apache.commons:commons-lang3:jar:3.3.2:compile
[INFO] |  +- org.apache.commons:commons-math3:jar:3.2:compile
[INFO] |  +- com.googlecode.concurrentlinkedhashmap:concurrentlinkedhashmap-lru:jar:1.4:compile
[INFO] |  +- org.antlr:antlr:jar:3.5.2:compile
[INFO] |  |  \- org.antlr:ST4:jar:4.0.8:compile
[INFO] |  +- org.antlr:antlr-runtime:jar:3.5.2:compile
[INFO] |  +- org.slf4j:slf4j-api:jar:1.7.12:compile
[INFO] |  +- org.slf4j:log4j-over-slf4j:jar:1.7.7:compile
[INFO] |  +- org.slf4j:jcl-over-slf4j:jar:1.7.5:compile
[INFO] |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.2:compile
[INFO] |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.2:compile
[INFO] |  +- com.googlecode.json-simple:json-simple:jar:1.1:compile
[INFO] |  +- com.boundary:high-scale-lib:jar:1.0.6:compile
[INFO] |  +- org.yaml:snakeyaml:jar:1.11:compile
[INFO] |  +- org.mindrot:jbcrypt:jar:0.3m:compile
[INFO] |  +- io.dropwizard.metrics:metrics-core:jar:3.1.0:compile
[INFO] |  +- com.addthis.metrics:reporter-config3:jar:3.0.0:compile
[INFO] |  |  +- com.addthis.metrics:reporter-config-base:jar:3.0.0:compile
[INFO] |  |  \- org.hibernate:hibernate-validator:jar:4.3.0.Final:compile
[INFO] |  |     +- javax.validation:validation-api:jar:1.0.0.GA:compile
[INFO] |  |     \- org.jboss.logging:jboss-logging:jar:3.1.0.CR2:compile
[INFO] |  +- com.thinkaurelius.thrift:thrift-server:jar:0.3.7:compile
[INFO] |  |  \- com.lmax:disruptor:jar:3.0.1:compile
[INFO] |  +- com.clearspring.analytics:stream:jar:2.5.2:compile
[INFO] |  |  \- it.unimi.dsi:fastutil:jar:6.5.7:compile
[INFO] |  +- net.sf.supercsv:super-csv:jar:2.1.0:compile
[INFO] |  +- ch.qos.logback:logback-core:jar:1.0.6:compile
[INFO] |  +- ch.qos.logback:logback-classic:jar:1.0.6:compile
[INFO] |  +- org.apache.thrift:libthrift:jar:0.9.2:compile
[INFO] |  |  +- org.apache.httpcomponents:httpclient:jar:4.2.5:compile
[INFO] |  |  \- org.apache.httpcomponents:httpcore:jar:4.2.4:compile
[INFO] |  +- org.apache.cassandra:cassandra-thrift:jar:2.2.4:compile
[INFO] |  +- net.java.dev.jna:jna:jar:4.0.0:compile
[INFO] |  +- com.github.jbellis:jamm:jar:0.3.0:compile
[INFO] |  +- com.github.tjake:crc32ex:jar:0.1.1:compile
[INFO] |  +- io.netty:netty-all:jar:4.0.23.Final:compile
[INFO] |  +- joda-time:joda-time:jar:2.4:compile
[INFO] |  \- org.fusesource:sigar:jar:1.6.4:compile
[INFO] +- com.google.guava:guava:jar:17.0:test
[INFO] +- com.datastax.cassandra:cassandra-driver-core:jar:3.1.2:compile
[INFO] |  +- io.netty:netty-handler:jar:4.0.37.Final:compile
[INFO] |  |  +- io.netty:netty-buffer:jar:4.0.37.Final:compile
[INFO] |  |  |  \- io.netty:netty-common:jar:4.0.37.Final:compile
[INFO] |  |  +- io.netty:netty-transport:jar:4.0.37.Final:compile
[INFO] |  |  \- io.netty:netty-codec:jar:4.0.37.Final:compile
[INFO] |  +- com.github.jnr:jnr-ffi:jar:2.0.7:compile
[INFO] |  |  +- com.github.jnr:jffi:jar:1.2.10:compile
[INFO] |  |  +- com.github.jnr:jffi:jar:native:1.2.10:runtime
[INFO] |  |  +- org.ow2.asm:asm:jar:5.0.3:compile
[INFO] |  |  +- org.ow2.asm:asm-commons:jar:5.0.3:compile
[INFO] |  |  +- org.ow2.asm:asm-util:jar:5.0.3:compile
[INFO] |  |  \- com.github.jnr:jnr-x86asm:jar:1.0.2:compile
[INFO] |  \- com.github.jnr:jnr-posix:jar:3.0.27:compile
[INFO] |     \- com.github.jnr:jnr-constants:jar:0.9.0:compile
[INFO] +- com.cisco.dascode:dascode-cassandra:zip:2.2.4:compile
[INFO] \- junit:junit:jar:4.11:test
[INFO]    \- org.hamcrest:hamcrest-core:jar:1.3:test

At first, I was using cassandra-unit to start up the server, but ran into a whole slew of other problems, so I ended up just starting the server from my code in a separate thread like this:

System.setProperty("cassandra.config", "file:" + file.getAbsolutePath()); // Path to cassandra.yaml
System.setProperty("cassandra-foreground", "true");
System.setProperty("cassandra.native.epoll.enabled", "false");
...
cassandraDaemon = new CassandraDaemon();
cassandraDaemon.activate();

The client code looks like this:

Cluster cluster = new       Cluster.Builder().addContactPoints("localhost").withPort(9042).withCredentials("cassandra", "cassandra").build();
Session session = cluster.connect("system_auth");

I've tried the connect() call both with and without the keyspace, and there's no difference in behavior. When the client tries to connect, I get the following exception:

com.datastax.driver.core.exceptions.AuthenticationException: Authentication error on host localhost/127.0.0.1:9042: Username and/or password are incorrect
at com.datastax.driver.core.Connection$8.apply(Connection.java:393)
at com.datastax.driver.core.Connection$8.apply(Connection.java:362)

The interesting thing is that if I start up the server part only, locally on my Mac, I can use the CQLSH client with username/password set to "cassandra"/"cassandra" and I don't have any issues with authentication.

My cassandra.yaml file looks as follows:

cluster_name: 'Test Cluster'
num_tokens: 256
hinted_handoff_enabled: true
hinted_handoff_throttle_in_kb: 1024
max_hints_delivery_threads: 2
batchlog_replay_throttle_in_kb: 1024
authenticator: PasswordAuthenticator
authorizer: CassandraAuthorizer
role_manager: CassandraRoleManager
roles_validity_in_ms: 2000
permissions_validity_in_ms: 2000
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
disk_failure_policy: stop
commit_failure_policy: stop
key_cache_size_in_mb:
key_cache_save_period: 14400
row_cache_size_in_mb: 0
row_cache_save_period: 0
counter_cache_size_in_mb:
counter_cache_save_period: 7200
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
seed_provider:
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
    parameters:
        - seeds: "127.0.0.1"
concurrent_reads: 32
concurrent_writes: 32
concurrent_counter_writes: 32
memtable_allocation_type: heap_buffers
index_summary_capacity_in_mb:
index_summary_resize_interval_in_minutes: 60
trickle_fsync: false
trickle_fsync_interval_in_kb: 10240
storage_port: 7000
ssl_storage_port: 7001
listen_address: localhost
start_native_transport: true
native_transport_port: 9042
start_rpc: false
rpc_address: localhost
rpc_port: 9160
rpc_keepalive: true
rpc_server_type: sync
thrift_framed_transport_size_in_mb: 15
incremental_backups: false
snapshot_before_compaction: false
auto_snapshot: true
tombstone_warn_threshold: 1000
tombstone_failure_threshold: 100000
column_index_size_in_kb: 64
batch_size_warn_threshold_in_kb: 5
batch_size_fail_threshold_in_kb: 50
compaction_throughput_mb_per_sec: 16
compaction_large_partition_warning_threshold_mb: 100
sstable_preemptive_open_interval_in_mb: 50

Any hints on what I might be doing wrong are appreciated.

Answer 1

I'm not sure if this will help, but it's possible that the cassandra user isn't present right away, if you add the following before creating your CassandraDaemon:

System.setProperty("cassandra.superuser_setup_delay_ms", "0");

This might fix your issue. The value defaults to 10000 (10 seconds) (source), which explains why sleeping for 20 seconds might have fixed your issue.

Answer 2

I guess it was a race condition. The server isn't fully initialized by the time the client tries to connect. I put a Thread.sleep(20000); in the code before trying to connect, and that worked. Is there a better way of verifying that the server is fully initialized?