CODEWITHSUNDEEP
cachingapache-sparkrddpersist
Edamame
Edamame

Reputation: 25366

Spark: How do I save an RDD before unpersist it

I want to unpersist myRDD after the data is saved. I have the following code:

  val isDone = myRDD.saveAsTextFile("myOutput")
  if (isDone != null) {
     myRDD.unpersist()
  }

but the line:

isDone != null

keeps saying: comparing values of types Unit and Null using `!=' will always yield true

What should be the correct way to solve this problem? Thank you!

Upvotes: 0

Views: 390

Answers (1)

Jason Scott Lenderman
Jason Scott Lenderman

Reputation: 1918

This should work fine:

myRDD.saveAsTextFile("myOutput")
myRDD.unpersist()

The data will be saved before the RDD is unpersisted.

Note that the saveAsTextFile method returns Unit. This is a type that is returned by any method (procedure) which does not return anything useful, and there is only one instance, i.e. (). So nothing useful is achieved by testing on the value returned by saveAsTextFile. Also, Unit being a subtype of AnyVal can never be equal to null, that's why you're getting the particular error that you're seeing. The same thing happens for Ints:

def foo(x: Int) = x != null

Upvotes: 1

Related Questions