CODEWITHSUNDEEP
scalaapache-sparkhbase
Jeevan
Jeevan

Reputation: 740

java.lang.ClassNotFoundException in scala program

I am trying to connect hbase from spark and I want to run scala jar file in spark-submit. Im not sure how to write classes in scala, can any one help

package com.jeevan.sparkhbase

import org.apache.spark._
import org.apache.spark.rdd.NewHadoopRDD
import org.apache.hadoop.hbase.{HBaseConfiguration, HTableDescriptor}
import org.apache.hadoop.hbase.client.HBaseAdmin
import org.apache.hadoop.hbase.mapreduce.TableInputFormat
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.HColumnDescriptor
import org.apache.hadoop.hbase.util.Bytes
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.HTable;



class InsertData {
  def main(arg: Array[String]) {
    val conf = HBaseConfiguration.create()
    val tableName = "emp"
    conf.set(TableInputFormat.INPUT_TABLE, tableName)

    val myTable = new HTable(conf, tableName);
    var p = new Put(new String("row999").getBytes());
    p.add("cf".getBytes(), "column_name".getBytes(), new String("value999").getBytes());
    myTable.put(p);
    myTable.flushCommits();
  }
}

I used maven to build jar and want to execute this jar file with spark-submit. following is the spark-submit command i used to run the jar

spark-submit --class com.jeevan.sparkhbase.InsertData --master local[*] SHIntegration-0.0.1-SNAPSHOT-jar-with-dependencies.jar

I am getting this error 

java.lang.ClassNotFoundException: com.jeevan.sparkhbase.InsertData
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.apache.spark.util.Utils$.classForName(Utils.scala:230)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:732)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)

can someone how to write this above code with class and object. appreciate your help

Upvotes: 2

Views: 4446

Answers (1)

Kit Menke
Kit Menke

Reputation: 7056

A couple things could be wrong here including how you packaged your jar.

First, InsertData should be an object not a class.

object InsertData {
  def main(arg: Array[String]) {
      // stuff
  }
}

Second, you aren't actually connecting to Spark anywhere. You'll need to add something like this in your app:

val spark = SparkSession.builder().appName(jobName).master("local[1]").getOrCreate()

Check out my spark-hello-world for a complete example project.

Upvotes: 1

Related Questions