CODEWITHSUNDEEP
cassandracassandra-3.0
dilsingi
dilsingi

Reputation: 2958

Cassandra auth issue while adding a new node

Whenever a new node is added in Apache Cassandra 3.9, we have a weird issue with system_auth keyspace. We do network topology replication with RF = number of nodes in cluster for the system_auth keyspace.

But after a new node bootstraps, this keyspace seems to be corrupted. Say a select on the roles table shows NULL data in certain rows. 

role                            | can_login | is_superuser | member_of | salted_hash
---------------------------------+-----------+--------------+-----------+--------------------------------------------------------------
                     user1      |      null |         null |      null |       null

"List users" command results in Java null pointer exception. Debug log shows the following info

ERROR [Native-Transport-Requests-3] 2017-06-27 15:17:02,927 ErrorMessage.java:349 - Unexpected exception during request
java.lang.NullPointerException: null
	at org.apache.cassandra.serializers.BooleanSerializer.deserialize(BooleanSerializer.java:33) ~[apache-cassandra-3.9.0.jar:3.9.0]
	at org.apache.cassandra.serializers.BooleanSerializer.deserialize(BooleanSerializer.java:24) ~[apache-cassandra-3.9.0.jar:3.9.0]
	at org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:115) ~[apache-cassandra-3.9.0.jar:3.9.0]
	at org.apache.cassandra.cql3.UntypedResultSet$Row.getBoolean(UntypedResultSet.java:273) ~[apache-cassandra-3.9.0.jar:3.9.0]
	at org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:85) ~[apache-cassandra-3.9.0.jar:3.9.0]
	at org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:81) ~[apache-cassandra-3.9.0.jar:3.9.0]
	at org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:503) ~[apache-cassandra-3.9.0.jar:3.9.0]
	at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:485) ~[apache-cassandra-3.9.0.jar:3.9.0]
	at org.apache.cassandra.auth.CassandraRoleManager.canLogin(CassandraRoleManager.java:298) ~[apache-cassandra-3.9.0.jar:3.9.0]
	at org.apache.cassandra.cql3.statements.ListUsersStatement.formatResults(ListUsersStatement.java:51) ~[apache-cassandra-3.9.0.jar:3.9.0]
	at org.apache.cassandra.cql3.statements.ListRolesStatement.resultMessage(ListRolesStatement.java:108) ~[apache-cassandra-3.9.0.jar:3.9.0]
	at org.apache.cassandra.cql3.statements.ListRolesStatement.execute(ListRolesStatement.java:85) ~[apache-cassandra-3.9.0.jar:3.9.0]
	at org.apache.cassandra.cql3.statements.AuthorizationStatement.execute(AuthorizationStatement.java:48) ~[apache-cassandra-3.9.0.jar:3.9.0]
	at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) ~[apache-cassandra-3.9.0.jar:3.9.0]
	at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:219) ~[apache-cassandra-3.9.0.jar:3.9.0]
	at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:204) ~[apache-cassandra-3.9.0.jar:3.9.0]
	at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115) ~[apache-cassandra-3.9.0.jar:3.9.0]
	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513) [apache-cassandra-3.9.0.jar:3.9.0]
	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:407) [apache-cassandra-3.9.0.jar:3.9.0]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.39.Final.jar:4.0.39.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366) [netty-all-4.0.39.Final.jar:4.0.39.Final]
	at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.39.Final.jar:4.0.39.Final]
	at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357) [netty-all-4.0.39.Final.jar:4.0.39.Final]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_131]
	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) [apache-cassandra-3.9.0.jar:3.9.0]
	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.9.0.jar:3.9.0]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]

Upvotes: 0

Views: 374

Answers (1)

dilsingi
dilsingi

Reputation: 2958

This seems to be a bug with Cassandra 3.9 which has been fixed in later version Cassandra 3.10. Here is the bug ticket https://issues.apache.org/jira/browse/CASSANDRA-12813

Upvotes: 1

Related Questions