CODEWITHSUNDEEP
javapythonscalamachine-learningapache-spark
David King
David King

Reputation: 39

Using Apache Spark MLib Libraries without Spark Console

I installed Apache Spark on Windows 7 and successfully got both the Scala and Python consoles to run..

My question is to know if using the console is necessary when simply wanting to utilize the Mlib (Machine Learning Library) directly from a Scala, Java or Python installation when not using the Spark Console.

Is there a simple way to do this, and are there major reasons why I might not want to do this?

Upvotes: 1

Views: 442

Answers (1)

Николай Митропольский
Николай Митропольский

Reputation: 917

Of course you don't have to use console. You can create Spark context in your code:

val conf = new SparkConf().setMaster("local[4]").setAppName("Spark Pi")
val spark = new SparkContext(conf)

then create RDD:

val data = spark.textFile("data/mllib/kmeans_data.txt")

and run Mlib algorithms on it.

Upvotes: 3

Related Questions