CODEWITHSUNDEEP
scalaapache-sparkapache-spark-ml
Georg Heiler
Georg Heiler

Reputation: 17694

Why does custom Estimator end up overriding Nothing type?

I am implementing a custom estimator which has troubles accessing a parameter ava.util.NoSuchElementException: Failed to find a default value for isInList. It is defined as follows:

trait PreprocessingParams extends Params {
  final val isInList = new Param[Array[String]](this, "isInList", "list of isInList items")
}

To better debug the problem I created a minimal example here https://gist.github.com/geoHeil/8dc7a6b6938a517f068e7fd6a981ed12 the ExampleTrans works just fine. However, I would rather like to include the functionality of the transformer into an estimator which performs some data cleaning as well.

But now I face strange compile issues overriding method has wrong type - expecting Nothing

What is wrong with the return types of my ExampleEstimator?

Upvotes: 3

Views: 65

Answers (1)

T. Gawęda
T. Gawęda

Reputation: 16086

You didn't specify the generic types of Estimator type constructor and so Nothing has been used instead.

Use:

class ExampleEstimator(override val uid: String) 
  extends Estimator[ExampleTransModel] with PreprocessingParams {
  ...
}

The full definition of Estimator is as follows:

abstract class Estimator[M <: Model[M]] extends PipelineStage

Note the generic type M that extends Model.

Upvotes: 3

Related Questions