Datastax : All host(s) tried for query failed (no host was tried)

Question

We are having 8 node datastax cluster, while starting to thrift server we are getting All host(s) tried for query failed (no host was tried)

ERROR 2017-12-30 21:16:16,317 org.apache.spark.deploy.DseSparkSubmitBootstrapper: Failed to start or submit Spark application
java.lang.RuntimeException: com.datastax.bdp.fs.model.InternalServerException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522) ~[hive-exec-1.2.1.spark2.jar:1.2.1.spark2]
        at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:189) ~[spark-hive_2.11-2.0.2.6-de611f9.jar:2.0.2.6-de611f9]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_151]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_151]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_151]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_151]
        at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:258) ~[spark-hive_2.11-2.0.2.6-de611f9.jar:2.0.2.6-de611f9]
        at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:359) ~[spark-hive_2.11-2.0.2.6-de611f9.jar:2.0.2.6-de611f9]
        at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:263) ~[spark-hive_2.11-2.0.2.6-de611f9.jar:2.0.2.6-de611f9]
        at org.apache.spark.sql.hive.HiveSharedState.metadataHive$lzycompute(HiveSharedState.scala:39) ~[spark-hive_2.11-2.0.2.6-de611f9.jar:2.0.2.6-de611f9]
        at org.apache.spark.sql.hive.HiveSharedState.metadataHive(HiveSharedState.scala:38) ~[spark-hive_2.11-2.0.2.6-de611f9.jar:2.0.2.6-de611f9]
        at org.apache.spark.sql.hive.HiveSessionState.metadataHive$lzycompute(HiveSessionState.scala:43) ~[spark-hive_2.11-2.0.2.6-de611f9.jar:2.0.2.6-de611f9]
        at org.apache.spark.sql.hive.HiveSessionState.metadataHive(HiveSessionState.scala:43) ~[spark-hive_2.11-2.0.2.6-de611f9.jar:2.0.2.6-de611f9]
        at org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:62) ~[spark-hive-thriftserver_2.11-2.0.2.6-de611f9.jar:2.0.2.6-de611f9]
        at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:81) ~[spark-hive-thriftserver_2.11-2.0.2.6-de611f9.jar:2.0.2.6-de611f9]
        at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala) ~[spark-hive-thriftserver_2.11-2.0.2.6-de611f9.jar:2.0.2.6-de611f9]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151]
        at org.apache.spark.deploy.DseSparkSubmit$.org$apache$spark$deploy$DseSparkSubmit$$runMain(DseSparkSubmit.scala:733) ~[dse-spark-5.1.5.jar:5.1.5]
        at org.apache.spark.deploy.DseSparkSubmit$.doRunMain$1(DseSparkSubmit.scala:177) ~[dse-spark-5.1.5.jar:5.1.5]
        at org.apache.spark.deploy.DseSparkSubmit$.submit(DseSparkSubmit.scala:202) ~[dse-spark-5.1.5.jar:5.1.5]
        at org.apache.spark.deploy.DseSparkSubmit$.main(DseSparkSubmit.scala:111) ~[dse-spark-5.1.5.jar:5.1.5]
        at org.apache.spark.deploy.DseSparkSubmitBootstrapper$.main(DseSparkSubmitBootstrapper.scala:116) ~[dse-spark-5.1.5.jar:5.1.5]
        at org.apache.spark.deploy.DseSparkSubmitBootstrapper.main(DseSparkSubmitBootstrapper.scala) [dse-spark-5.1.5.jar:5.1.5]
Caused by: com.datastax.bdp.fs.model.InternalServerException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)
        at com.datastax.bdp.fs.model.DseFsJsonProtocol$ThrowableReader$.read(DseFsJsonProtocol.scala:236) ~[dse-dsefs-common-5.1.5.jar:5.1.5]
        at com.datastax.bdp.fs.model.DseFsJsonProtocol$ThrowableReader$.read(DseFsJsonProtocol.scala:211) ~[dse-dsefs-common-5.1.5.jar:5.1.5]
        at spray.json.JsValue.convertTo(JsValue.scala:31) ~[spray-json_2.11-1.3.2.jar:na]
        at com.datastax.bdp.fs.rest.RestResponse$stateMachine$macro$281$1.apply(RestResponse.scala:48) ~[dse-dsefs-common-5.1.5.jar:5.1.5]
        at com.datastax.bdp.fs.rest.RestResponse$stateMachine$macro$281$1.apply(RestResponse.scala:44) ~[dse-dsefs-common-5.1.5.jar:5.1.5]
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36) ~[scala-library-2.11.11.jar:na]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:408) ~[netty-all-4.0.42.Final.jar:4.0.42.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:455) ~[netty-all-4.0.42.Final.jar:4.0.42.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) ~[netty-all-4.0.42.Final.jar:4.0.42.Final]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_151]
ERROR 2017-12-30 21:16:17,048 org.apache.spark.scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorAdded(1514668577045,1,org.apache.spark.scheduler.cluster.ExecutorData@adac7383)
ERROR 2017-12-30 21:16:17,108 org.apache.spark.scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerBlockManagerAdded(1514668577108,BlockManagerId(1, , 40533),384093388)

Answer 1

The error is being thrown by the Metastore attempting to setup a directory in DSEFS and failing. We can see the DSEFS root here

java.lang.RuntimeException: com.datastax.bdp.fs.model.InternalServerException:
  com.datastax.driver.core.exceptions.NoHostAvailableException: 
    All host(s) tried for query failed (no host was tried)

The most common cause of this is that the Replication on the DSEFS based key-space is not configured such that a LOCAL consistency request could not succeed from the node in question. I would check the replication and make sure that if you are using NetworkTopologyStrategy that the DC name matches the current DC. If you have multiple DC's and are using SimpleStrategy this would also cause breaks.