CODEWITHSUNDEEP
scalaapache-spark
Thangarajan Pannerselvam
Thangarajan Pannerselvam

Reputation: 146

Spark Scala Flumeutill not found

I am getting the below error:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/streaming/flume/FlumeUtils$ at SampleStream$.main(NetworkWordCount.scala:10) at SampleStream.main(NetworkWordCount.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:665) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.ClassNotFoundException: org.apache.spark.streaming.flume.FlumeUtils$ at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

i refer below link

https://spark.apache.org/docs/1.4.1/streaming-flume-integration.html

Upvotes: 0

Views: 474

Answers (2)

zsxwing
zsxwing

Reputation: 20816

You need to package spark-streaming-flume and its dependencies (except spark-core and spark-streaming which are provided by spark-submit) into the application JAR.

https://spark.apache.org/docs/1.6.1/streaming-flume-integration.html has a better description about deploying. You can use this doc as there is no API changes between 1.4 and 1.6.

Upvotes: 0

gabriel9
gabriel9

Reputation: 86

You need to have all libraries loaded in classpath. Add this to build.sbt file: libraryDependencies += "org.apache.spark" % "spark-streaming-flume_2.10" % "1.6.1"

Upvotes: 0

Related Questions