CODEWITHSUNDEEP
rapache-sparksparkr
lord12
lord12

Reputation: 2927

SparkR Null Pointer Exception when trying to create a data frame

When trying to create a data frame in sparkR, I get an error regarding a Null Pointer Exception. I have pasted my code, and the error message below. Do I need to install any more packages in order for this code to run?

CODE

SPARK_HOME <- "C:\\Users\\erer\\Downloads\\spark-1.5.2-bin-hadoop2.4\\spark-1.5.2-bin-hadoop2.4"
Sys.setenv('SPARKR_SUBMIT_ARGS'='"--packages" "com.databricks:spark-csv_2.10:1.2.0" "sparkr-shell"')
library(SparkR, lib.loc = "C:\\Users\\erer\\Downloads\\spark-1.5.2-bin-hadoop2.4\\R\\lib")
    library(SparkR)
        library(rJava)

        sc <- sparkR.init(master = "local", sparkHome = SPARK_HOME)
        sqlContext <- sparkRSQL.init(sc)

        localDF <- data.frame(name=c("John", "Smith", "Sarah"), age=c(19, 23, 18))
        df <- createDataFrame(sqlContext, localDF)

ERROR:

Error in invokeJava(isStatic = FALSE, objId$id, methodName, ...) : 
  org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1, localhost): java.lang.NullPointerException

        at java.lang.ProcessBuilder.start(Unknown Source)

        at org.apache.hadoop.util.Shell.runCommand(Shell.java:445)

        at org.apache.hadoop.util.Shell.run(Shell.java:418)

        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)

        at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:873)

        at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:853)

        at org.apache.spark.util.Utils$.fetchFile(Utils.scala:381)

        at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:405)

        at org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Executor$$updateDependencies$5.apply(Executor.scala:397)

        at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:7

Upvotes: 1

Views: 474

Answers (1)

KrisP
KrisP

Reputation: 1216

You need to point library SparkR to the directory where the local SparkR code is, specified in the lib.loc parameter (if you downloaded a Spark binary, the SPARK_HOME/R/lib will be already populated for you): 

`library(SparkR, lib.loc = "/home/kris/spark/spark-1.5.2-bin-hadoop2.6/R/lib")`

See also this tutorial on R-bloggers on how to run Spark from Rstudio: http://www.r-bloggers.com/sparkr-with-rstudio-in-ubuntu-12-04/

Upvotes: 1

Related Questions