CODEWITHSUNDEEP
javamavenhadoopapache-spark
New Man
New Man

Reputation: 229

Run Spark over Hadoop

In my local machine, I have installed scala, spark, hadoop. I have done one sample program of word count using spark with maven. Now I have to do these things over Hadoop using spark?. How can I do this??

Upvotes: 1

Views: 159

Answers (1)

mgaido
mgaido

Reputation: 3055

Here you can find the docs about how to run Spark on YARN. This means that Spark will use the same resource manager of your Hadoop distribution.

If, instead, you meant only that you want Spark to be run using files stored through HDFS, than you can simply run Spark in standalone mode and specify the hdfs:// URL of the files he has to deal with. In this case, you just have to set the Spark master to local and run your application. Instruction about how to install and submit applications in standalone mode are here

Upvotes: 2

Related Questions