CODEWITHSUNDEEP
javaapache-sparkapache-spark-sqlapache-spark-mllibapache-spark-ml
ngi
ngi

Reputation: 51

Can we create custom Estimators

I want to create my own Estimator for Spark ml pipeline purpose so that I can use my own custom business logic.

If any one can guide me in this using Java will be very helpful.

Update:
I created one Estimator after Matt suggestion but not sure I am doing in proper way or not please suggest.

public class CustomPredictors extends Predictor<Vector, LogisticRegression, LogisticRegressionModel> {

    private static final long serialVersionUID = -6860916159403147252L;
    
    private String uid;

    public String getUid() {
        if(uid==null) {
            uid=Identifiable.randomUID("CustomPredictors");
        }
        return uid;
    }

    @Override
    public String uid() {
        return getUid();
    }

    public static MLReader<CustomPredictors> read() {
        return new DefaultParamsReader<>();
    }

    @Override
    public LogisticRegression copy(ParamMap extra) {
        return defaultCopy(extra);
    }

    @Override
    public LogisticRegressionModel train(Dataset<?> dataset) {
        System.out.println("CustomPredictors");
        //TODO put business logic
        return null;
    }   
}

Upvotes: 0

Views: 39

Answers (1)

Matt Andruff
Matt Andruff

Reputation: 5125

Most of the code in spark is written in Scala. All of it compiles down to java, so you can totally write Java, just might be a little translation.

You can make your own custom Estimator class. Here's an example of one that's been created in Scala.

Upvotes: 1

Related Questions