CODEWITHSUNDEEP
spark-streamingflume
G.Saleh
G.Saleh

Reputation: 509

not found: value FlumeUtils

This is my code for integration of spark streaming with flume :

    val conf = new SparkConf()
      .setAppName("File Count")
      .setMaster("local[2]")

    val sc = new SparkContext(conf)
    val ssc = new StreamingContext(sc, Seconds(10))
val flumeStream = FlumeUtils.createPollingStream(ssc,192.168.1.31,8020)

But i have an error : not found: value FlumeUtils
This is my pom.xml :

        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>2.10.4</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.10</artifactId>
            <version>1.6.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-flume-sink_2.10</artifactId>
            <version>1.5.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.10</artifactId>
            <version>1.5.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-lang3</artifactId>
            <version>3.3.2</version>
        </dependency>

My spark version is 1.5.0
Any help !!! and thanks in advance.

Upvotes: 0

Views: 381

Answers (1)

sachin thirumala
sachin thirumala

Reputation: 51

FlumeUtils is a class under org.apache.spark.streaming.flume. In your pom.xml, the artifact you are trying to import is spark-streaming-flume-sink_2.10 which doesn't have the FlumeUtils class.

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-flume-sink_2.10</artifactId>
    <version>1.5.0</version>
</dependency>

Instead import the below artifact:

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-flume_2.10</artifactId>
    <version>1.5.0</version>
</dependency>

Hope this helps.

Upvotes: 1

Related Questions