CODEWITHSUNDEEP
javacassandratimeoutastyanax
Steffen Harbich
Steffen Harbich

Reputation: 2759

Astyanax not respecting configured socket timeout

This is my setup:

I am setting configuration like

AstyanaxContext<Keyspace> context = new AstyanaxContext.Builder()
    .forCluster(service.getClusterName())
    .forKeyspace(service.getKeySpaceName())
    .withAstyanaxConfiguration(new AstyanaxConfigurationImpl()
         .setDiscoveryType(NodeDiscoveryType.NONE)
         .setCqlVersion("3.0.0")
         .setDefaultReadConsistencyLevel(consistencyLevel.getAstyanaxValue())
         .setDefaultWriteConsistencyLevel(consistencyLevel.getAstyanaxValue())
    )
    .withConnectionPoolConfiguration(new ConnectionPoolConfigurationImpl("b2bConnectionPool")
         .setPort(service.getPort())
         .setMaxConnsPerHost(5)
         .setSeeds(StringUtils.join(hosts, ","))
         // increase default timeout for heavy operations (milliseconds)
         .setSocketTimeout(15000)
         .setSSLConnectionContext(sslContext)
         .setAuthenticationCredentials(credentials)
    )
    .withConnectionPoolMonitor(new CountingConnectionPoolMonitor())
    .buildKeyspace(ThriftFamilyFactory.getInstance());

Now there is a reproducible query that takes a long time and finally throws a OperationTimeoutException:

com.netflix.astyanax.connectionpool.exceptions.OperationTimeoutException: OperationTimeoutException: [host=myhost(myip):13260, latency=10001(40007), attempts=4]TimedOutException()
    at com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:171) ~[astyanax-thrift-1.56.49.jar:na]
    at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:65) ~[astyanax-thrift-1.56.49.jar:na]
    at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$2.execute(ThriftColumnFamilyQueryImpl.java:190) ~[astyanax-thrift-1.56.49.jar:na]
    at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$2.execute(ThriftColumnFamilyQueryImpl.java:182) ~[astyanax-thrift-1.56.49.jar:na]
    at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.execute(ThriftSyncConnectionFactoryImpl.java:151) ~[astyanax-thrift-1.56.49.jar:na]
    at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:119) ~[astyanax-core-1.56.49.jar:na]
    at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:338) ~[astyanax-core-1.56.49.jar:na]
    at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1.execute(ThriftColumnFamilyQueryImpl.java:180) ~[astyanax-thrift-1.56.49.jar:na]

The exception message says "latency=10001" and I thought this should be the socket timeout configured to 15000 ms but it's obviously not. How can I increase the timeout for a query operation in astyanax?

Upvotes: 4

Views: 302

Answers (2)

Abhishek Garg
Abhishek Garg

Reputation: 2288

I was getting the similar timeout exception when trying to query nodes with bigger data. I modified these 4 values in cassandra.yaml and it resolved all the timeout errors.

How long the coordinator should wait for read operations to complete read_request_timeout_in_ms: 15000

How long the coordinator should wait for seq or index scans to complete range_request_timeout_in_ms: 30000

How long the coordinator should wait for writes to complete write_request_timeout_in_ms: 30000

How long the coordinator should wait for counter writes to complete counter_write_request_timeout_in_ms: 15000

Note: You need to do this in all the nodes in the cluster and also need to restart Cassandra in all of them.

Upvotes: 1

Ajay Deshwal
Ajay Deshwal

Reputation: 1306

I believe you are not encountering a socket timeout, rather an rpc timeout which i think cannot be controlled through Astyanax. In your stack trace, the exception wrapped inside OperationTimeoutException is TimedOutException which is thrown when an rpc timeout occurs (default 10 seconds). In case of a socket timeout, the SocketTimeoutException will be thrown.

Try executing your query with cqlsh and you will get a Request did not complete within rpc_timeout message.

Inside your <cassandra_home>/conf/cassandra.yaml try setting:

read_request_timeout_in_ms: 20000
rpc_timeout_in_ms: 20000

Restart your server and try the query execution again.

Upvotes: 1

Related Questions