CODEWITHSUNDEEP
scalacsvapache-sparkapache-spark-mllib
Shishir Anshuman
Shishir Anshuman

Reputation: 1125

export als recommendation model to a file

I am new to Apache Spark. I ran the sample ALS algorithm code present in the examples folder. I gave a csv file as an input. When I use model.save(path) to save the model, it is stored in gz.parquet file.

When I tried to open this file, I get these errors

Now I want to store the recommendation model generated in a text or csv file for using it outside Spark.

I tried the following function to store the model generated in a file but it was useless:

model.saveAsTextFile("path")

Please suggest me a way to overcome this issue.

Upvotes: 0

Views: 1323

Answers (2)

Shishir Anshuman
Shishir Anshuman

Reputation: 1125

As it turns out saveAsTextFile() only works on the slaves.Use collect() to collect the data from the slaves so it can be saved locally on the master. Solution can be found here

Upvotes: 0

JoseM LM
JoseM LM

Reputation: 373

Lest say you have trained your model with something like this:

val model = ALS.train(ratings, rank, numIterations, 0.01)

All that you have to do is:

import org.apache.spark.mllib.recommendation.ALS
import org.apache.spark.mllib.recommendation.MatrixFactorizationModel
import org.apache.spark.mllib.recommendation.Rating    
// Save
model.save(sc, "yourpath/yourmodel")
// Load Model
val sameModel = MatrixFactorizationModel.load(sc, "yourpath/yourmodel")

Upvotes: 1

Related Questions