A N
Reputation: 53
Exception in thread "main" com.datastax.driver.core.exceptions.ServerError: An unexpected error occurred server side on
While trying to fetch around 500000 records from cassandra using the paging support, I observe the following exception some times. Please note that its an intermittent exception and not seen every time I run my program.
Cassandra version : 3.10 Datastax driver : 3.1.0
Statement statement = new SimpleStatement("select * from mytable");
rs = session.execute(stmt);
for (Row row : rs) {
if (rs.getAvailableWithoutFetching() == 100) {
rs.fetchMoreResults();
}
// Do something with row
}
Sometimes after fetching around 200000 records the program fails with below exception :
Exception in thread "main" com.datastax.driver.core.exceptions.ServerError: An unexpected error occurred server side on myhost:9042: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Invalid metadata has been detected for role myuser
at com.datastax.driver.core.exceptions.ServerError.copy(ServerError.java:63)
at com.datastax.driver.core.exceptions.ServerError.copy(ServerError.java:25)
at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
at com.datastax.driver.core.ArrayBackedResultSet$MultiPage.prepareNextRow(ArrayBackedResultSet.java:312)
at com.datastax.driver.core.ArrayBackedResultSet$MultiPage.isExhausted(ArrayBackedResultSet.java:269)
at com.datastax.driver.core.ArrayBackedResultSet$1.hasNext(ArrayBackedResultSet.java:143)
Caused by: com.datastax.driver.core.exceptions.ServerError: An unexpected error occurred server side on myhost:9042: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Invalid metadata has been detected for role myuser
at com.datastax.driver.core.Responses$Error.asException(Responses.java:108)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:500)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1012)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:935)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1280)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:890)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:564)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:505)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:419)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:391)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
at java.lang.Thread.run(Thread.java:748)
Cassandra logs :
WARN [Native-Transport-Requests-1] 2018-01-09 10:07:31,832 CassandraRoleManager.java:96 - An invalid value has been detected in the roles table for role myuser. If you are unable to login, you may need to disable authentication and confirm that values in that table are accurate
ERROR [Native-Transport-Requests-1] 2018-01-09 10:07:31,833 QueryMessage.java:129 - Unexpected error during query
com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Invalid metadata has been detected for role myuser
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na]
at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.service.ClientState.authorize(ClientState.java:419) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:352) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:316) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:300) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:221) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:214) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:248) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:233) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517) [apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.10.jar:3.10]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.39.Final.jar:4.0.39.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366) [netty-all-4.0.39.Final.jar:4.0.39.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.39.Final.jar:4.0.39.Final]
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357) [netty-all-4.0.39.Final.jar:4.0.39.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_152]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.10.jar:3.10]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_152]
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Invalid metadata has been detected for role myuser
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na]
at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.RolesCache.getRoles(RolesCache.java:44) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:51) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:71) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:81) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.PermissionsCache.lambda$new$0(PermissionsCache.java:37) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:183) ~[apache-cassandra-3.10.jar:3.10]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
... 26 common frames omitted
Caused by: java.lang.RuntimeException: Invalid metadata has been detected for role myuser
at org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:99) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:82) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:528) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:503) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:281) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.RolesCache.lambda$new$0(RolesCache.java:36) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:183) ~[apache-cassandra-3.10.jar:3.10]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
... 40 common frames omitted
Caused by: java.lang.NullPointerException: null
at org.apache.cassandra.cql3.UntypedResultSet$Row.getBoolean(UntypedResultSet.java:273) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:88) ~[apache-cassandra-3.10.jar:3.10]
... 50 common frames omitted
ERROR [Native-Transport-Requests-1] 2018-01-09 10:07:31,834 ErrorMessage.java:384 - Unexpected exception during request
com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Invalid metadata has been detected for role myuser
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na]
at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.service.ClientState.authorize(ClientState.java:419) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:352) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:316) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:300) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:221) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:214) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:248) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:233) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517) [apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.10.jar:3.10]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.39.Final.jar:4.0.39.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366) [netty-all-4.0.39.Final.jar:4.0.39.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.39.Final.jar:4.0.39.Final]
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357) [netty-all-4.0.39.Final.jar:4.0.39.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_152]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.10.jar:3.10]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_152]
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Invalid metadata has been detected for role myuser
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na]
at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.RolesCache.getRoles(RolesCache.java:44) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:51) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:71) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:81) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.PermissionsCache.lambda$new$0(PermissionsCache.java:37) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:183) ~[apache-cassandra-3.10.jar:3.10]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
... 26 common frames omitted
Caused by: java.lang.RuntimeException: Invalid metadata has been detected for role myuser
at org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:99) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:82) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:528) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:503) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:281) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.RolesCache.lambda$new$0(RolesCache.java:36) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:183) ~[apache-cassandra-3.10.jar:3.10]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
... 40 common frames omitted
Caused by: java.lang.NullPointerException: null
at org.apache.cassandra.cql3.UntypedResultSet$Row.getBoolean(UntypedResultSet.java:273) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:88) ~[apache-cassandra-3.10.jar:3.10]
... 50 common frames omitted
INFO [Native-Transport-Requests-1] 2018-01-09 10:08:06,156 Message.java:617 - Unexpected exception during request; channel = [id: 0xbbaa062e, L:/127.0.0.1:9042 ! R:/127.0.0.1:9042]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer
at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.39.Final.jar:4.0.39.Final]
Upvotes: 1
Views: 3391
Answers (1)
Prashant Bhagat
Reputation: 21
I also faced the same problem. Looks like this error comes when system_auth.roles table across node is not in sync. To resolved this error, i performed below steps and it got resolved.
- select * from system_auth.roles where role = 'The role which is throwing error';
- nodetool repair system_auth; #on all nodes one after another
- consistency QUORUM;
- select * from system_auth.roles; #Read all roles once
- list roles;
Upvotes: 2
Related Questions
- Getting error as java.lang.NoClassDefFoundError:com/datastax/driver/core/policies/SpeculativeExecutionPolicy
- cassandra datastax Unexpected error while writing generated code: java.lang.NullPointerException
- com.datastax.driver.core.exceptions.BusyPoolException
- DataStax java.lang.AssertionError: Invalid offset
- Cassandra java driver connection error
- java cassandra connection java.lang.ClassNotFoundException: com.datastax.driver.core.Cluster
- Exception when connecting to Cassandra with CQL using DataStax Java driver 1.0.4
- DataStax Java driver 2.0.0-beta2 with Cassandra 1.2.1: unsupported protocol version
- com.datastax.driver.core.exceptions.InvalidQueryException using Datastax Java driver
- Cassandra 1.2.6 java datastax driver EXCEPTION No handler set for stream 0