CODEWITHSUNDEEP
hiveapache-spark
facha
facha

Reputation: 12502

executing Hive queries from Spark

I'm trying to execute this code in spark-shell:

val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
sqlContext.sql("show tables")

After executing the second line, I'm getting an exception:

java.lang.NoSuchMethodError: org.apache.hadoop.hive.ql.Driver.getResults(Ljava/util/ArrayList;)Z at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:305) at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:272) at org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult$lzycompute(NativeCommand.scala:35) at org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult(NativeCommand.scala:35) at org.apache.spark.sql.hive.execution.NativeCommand.execute(NativeCommand.scala:38) at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(HiveContext.scala:360) at org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.scala:360) at org.apache.spark.sql.SchemaRDDLike$class.$init$(SchemaRDDLike.scala:58) at org.apache.spark.sql.SchemaRDD.(SchemaRDD.scala:103) at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:98) at $iwC$$iwC$$iwC$$iwC.(:14) ...

Please, help me resolve this.

Upvotes: 0

Views: 12649

Answers (2)

Vinay Kumar Dudi
Vinay Kumar Dudi

Reputation: 137

You can use spark-sql.

To run hive queries:

spark-sql --master yarn --num-executors 6 --executor-memory 2G --executor-cores 1 --driver-memory 5G -e "select * from database.tablename;"

To run hive scripts:

spark-sql --master yarn --num-executors 6 --executor-memory 2G --executor-cores 1 --driver-memory 5G -f hivescript.hql

Note: Make sure hive is configured with spark installation.

Upvotes: 3

Mike Park
Mike Park

Reputation: 10931

You likely have a version of the Hive libraries that isn't compatible with your Spark libraries. Spark is expecting the function org.apache.hadoop.hive.ql.Driver.getResults(Ljava/util/ArrayList;) to exist, but in your Hive libraries, it doesn't.

Upvotes: 1

Related Questions