CODEWITHSUNDEEP
rapache-sparksparkr
Marcelo Pires
Marcelo Pires

Reputation: 61

SparkR Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState'

I am installing SparkR in my Windows 8.1 from this tutorial https://www.linkedin.com/pulse/setting-up-sparkr-windows-machine-ramabhadran-kapistalam. I ended it so I guess it's well implemented. The problem is when I try to run an example with a simple Data Frame:

Error in handleErrors(returnStatus, conn) : 
  java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':
    at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:981)
    at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:110)
    at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:109)
    at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:62)
    at org.apache.spark.sql.SparkSession.createDataFrame(SparkSession.scala:552)
    at org.apache.spark.sql.SparkSession.createDataFrame(SparkSession.scala:307)
    at org.apache.spark.sql.api.r.SQLUtils$.createDF(SQLUtils.scala:139)
    at org.apache.spark.sql.api.r.SQLUtils.createDF(SQLUtils.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at

This is my code in R:

 Sys.setenv(SPARK_HOME = "C:/Spark/spark-2.1.1-bin-hadoop2.7")
 .libPaths(c(file.path(Sys.getenv("SPARK_HOME"),"R","lib"), .libPaths()))
 sparkR.session(appName = "SparkR-DataFrame-example")
 df <- as.DataFrame(faithful)

And i saw a solution that I had to configure the sparkr session by adding:

sparkR.session(master = "local[*]", sparkConfig = list(spark.driver.memory = "1g", spark.sql.warehouse.dir = "file:///somelocaldirectory"))

I tried to edit the spark.sql.warehouse.dir with a data file but the error persisted

Upvotes: 4

Views: 2276

Answers (1)

muon
muon

Reputation: 14037

I had exact same problem, starting R-studio or R as administrator solved this issue.

to start as administrator, right click on R-studio or R and select run as administrator and then your commands should work just fine.

Upvotes: 2

Related Questions