CODEWITHSUNDEEP
apache-sparkcloudera-cdhhivecontext
陈志明
陈志明

Reputation: 21

CDH5.4.2 spark can use HiveContent in spark-shell but can't open spark-sql

I am using spark(standalone) of CDH5.4.2

After copying hive-site.xml to $SPARK_HOME/conf,i can query from hive in spark-shell,such as below:

scala> val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc); hiveContext: org.apache.spark.sql.hive.HiveContext = org.apache.spark.sql.hive.HiveContext@6c6f3a15 scala> hiveContext.sql("show tables").show();

But when i open spark-sql ,it show wrong:

java.lang.ClassNotFoundException: org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver Failed to load main class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.

You need to build Spark with -Phive and -Phive-thriftserver.

What is different between spark-shell and spark-sql? If spark of cdh don't support hive,why can i use HiveConext?

Upvotes: 0

Views: 773

Answers (1)

Oleg Chirukhin
Oleg Chirukhin

Reputation: 1000

Cloudera has a list of unsupported features here:

https://docs.cloudera.com/runtime/7.2.6/spark-overview/topics/spark-unsupported-features.html

The Thrift server is not supported.

This is a copy of the list for 7.2.6:

  • Apache Spark experimental features/APIs are not supported unless stated otherwise.
  • Using the JDBC Datasource API to access Hive or Impala is not supported
  • ADLS not supported for All Spark Components. Microsoft Azure Data Lake Store (ADLS) is a cloud-based filesystem that you can access through Spark applications. Spark with Kudu is not currently supported for ADLS data. (Hive on Spark is available for ADLS.)
  • IPython / Jupyter notebooks is not supported. The IPython notebook system (renamed to Jupyter as of IPython 4.0) is not supported.
  • Certain Spark Streaming features, such as the mapWithState method, are not supported.
  • Thrift JDBC/ODBC server is not supported
  • Spark SQL CLI is not supported
  • GraphX is not supported
  • SparkR is not supported
  • Structured Streaming is supported, but the following features of it are not:
  • Continuous processing, which is still experimental, is not supported.
  • Stream static joins with HBase have not been tested and therefore are not supported.
  • Spark cost-based optimizer (CBO) not supported.

Upvotes: 0

Related Questions