CODEWITHSUNDEEP
scalaapache-sparkapache-spark-mllib
Pranav
Pranav

Reputation: 33

How to save results of Model to text file?

I am trying to save the Frequent itemsets generated from the model to a text file. The code is an example of FPGrowth example in Spark ML library.

Using saveAsTextFile directly on the model writes the RDD locations and not the actual values.

import org.apache.spark.mllib.fpm.FPGrowth
import org.apache.spark.rdd.RDD

val data = sc.textFile("/home/ponny/Freq")
val data1 = sc.textFile("/home/ponny/Scala_Examples/test.txt")
val transactions: RDD[Array[String]] = data.map(s => s.trim.split(' '))
val tdata: RDD[Array[String]] = data1.map(s => s.trim.split(' '))

val fpg = new FPGrowth().setMinSupport(0.0).setNumPartitions(10)
val model = fpg.run(transactions)

model.freqItemsets.collect().foreach { itemset =>
  println(itemset.items.mkString("[", ",", "]") + ", " + itemset.freq)
}
model.freqItemsets.saveAsTextFile("/home/ponny/Freq_op")

The output generated in text file is like

org.apache.spark.mllib.fpm.FPGrowth$FreqItemset@5b27c9
org.apache.spark.mllib.fpm.FPGrowth$FreqItemset@2a7acd
org.apache.spark.mllib.fpm.FPGrowth$FreqItemset@d4d011
org.apache.spark.mllib.fpm.FPGrowth$FreqItemset@1fd4350

Please guide.

Upvotes: 3

Views: 1342

Answers (1)

zero323
zero323

Reputation: 330173

Exactly the same way how you print the values. Build desired output string first:

model.freqItemsets
  .map { fi => s"""[${fi.items.mkString(",")}], ${fi.freq}""" }
  .saveAsTextFile(path)

Upvotes: 3

Related Questions