CODEWITHSUNDEEP
apache-sparkmachine-learning
Dmitry
Dmitry

Reputation: 355

Get Spark metrics on each iteration step?

Applying spark's logistic regression on a specific dataset requires to define a number of iterations. So far I've learned that outputting the result of the cost function on each iteration might be useful information to plot. It can be used to visualize how many iterations a function needs to converge to a minimum. I was wondering if there is a way to output such information in spark? Looping over a train() function with different iteration numbers, sounds like a solution that requires a lot of time on large datasets. It would be nice to know if there is a better one already built in. Thanks for any advice on this topic.

Upvotes: 0

Views: 1185

Answers (1)

Prune
Prune

Reputation: 77857

After you've trained a model (call it myModel) that has such a history, you can get the iteration-by-iteration history with

myModel.summary.objectiveHistory.foreach(...)

There's a nice example here in the Spark ML documentation -- once you know the right search terms.

Upvotes: 2

Related Questions