CODEWITHSUNDEEP
scalahadoopapache-spark
gprivitera
gprivitera

Reputation: 943

HDFS Directory as parameter in Spark Streaming

I'm having problems with the Spark Streaming example: https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/streaming/examples/HdfsWordCount.scala

When I try to launch it using SBT

run local /user/dir/subdir/

I get this exception

[info] Running org.apache.spark.streaming.examples.HdfsWordCount local /user/dir/subdir/
14/04/21 18:45:55 INFO StreamingExamples: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
14/04/21 18:45:55 INFO StreamingExamples: Setting log level to [WARN] for streaming example. To override add a custom log4j.properties to the classpath.
14/04/21 18:45:55 WARN Utils: Your hostname, ubuntu resolves to a loopback address: 127.0.1.1; using 10.4.4.6 instead (on interface eth0)
14/04/21 18:45:55 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
14/04/21 18:45:57 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/04/21 18:46:00 ERROR JobScheduler: Error generating jobs for time 1398098760000 ms
java.io.FileNotFoundException: File /user/dir/subdir/ does not exist

I'm sure the directory exists on Hadoop fs and I even copied a file there. Is there some kind of input formatting I'm not aware of?

Upvotes: 0

Views: 1516

Answers (2)

David
David

Reputation: 362

You must take a look at your core-site.xml file inside hadoop. It must have a property with the default path

<configuration>
    <property>
          <name>fs.default.name</name>
          <value>hdfs://localhost:9000</value>
    </property>
</configuration>

Upvotes: 0

gprivitera
gprivitera

Reputation: 943

I've found the solution to my answer. The correct way to input a hdfs directory is this, atleast in my case:

run local hdfs://localhost:9000/user/dir/subdir/

I've found this in Spark Documentation: http://spark.apache.org/docs/latest/spark-standalone.html#running-alongside-hadoop

Upvotes: 1

Related Questions