CODEWITHSUNDEEP
scalaapache-sparkapache-spark-sql
user3685285
user3685285

Reputation: 6586

scala Unspecified Value Parameters

I want to extend the SparkSession class in spark. I copied the constructor of the original SparkSession partially reproduced here:

class SparkSession private(
    @transient val sparkContext: SparkContext,
    @transient private val existingSharedState: Option[SharedState],
    @transient private val parentSessionState: Option[SessionState],
    @transient private[sql] val extensions: SparkSessionExtensions)
  extends Serializable with Closeable with Logging { self =>

  private[sql] def this(sc: SparkContext) {
    this(sc, None, None, new SparkSessionExtensions)
  }

  // other implementations

}

Here's my attempt at extending it:

class CustomSparkSession private(
    @transient override val sparkContext: SparkContext,
    @transient private val existingSharedState: Option[SharedState],
    @transient private val parentSessionState: Option[SessionState],
    @transient override private[sql] val extensions: SparkSessionExtensions)
  extends SparkSession {

  // implementation

}

But I get an error on the SparkSession part of extends SparkSession with error:

Unspecified value parameters: sc: SparkContext

I know that it's coming from the this constructor in the original SparkContext, but I'm not sure how, or if I can even extend this properly. Any ideas?

Upvotes: 3

Views: 14033

Answers (1)

Dima
Dima

Reputation: 40500

When you write class Foo extends Bar you are actually (1) creating a default (no-argument) constructor for class Foo, and (2) calling a default constructor of class Bar.

Consequently, if you have something like class Bar(bar: String), you can't just write class Foo extends Bar, because there is no default constructor to call, you need to pass a parameter for bar. So, you could write something like

class Foo(bar: String) extends Bar(bar)

This is why you are seeing this error - you are trying to call a constructor for SparkSession, but not passing any value for sc.

But you have a bigger problem. That private keyword you see next to SparkSession (and another one before this) means that the constructor is ... well ... private. You cannot call it. In other words, this class cannot be subclassed (outside the sql package), so you should look for another way to achieve what you are trying to do.

Upvotes: 15

Related Questions