CODEWITHSUNDEEP
scalaapache-sparklog4j
a.moussa
a.moussa

Reputation: 3287

catch an error on spark

I use a code like this to execute some transformation and to continue the process even if I have bad value:

import scala.util.Try

val data = Array(Seq(1, 2), Seq(3, 4, 5))
val rdd = sc.parallelize(data)
val rdd_test = rdd.flatMap(x => Try(x(2)).toOption)
rdd_test.collect.foreach(println)

this simple code return 5 because in my first line, I don't have any value at third position. My question is how to catch the first line in order to use log4j to log an alert message. I want to say something like: Index Error. Do you have any idea?

Upvotes: 1

Views: 544

Answers (1)

jamborta
jamborta

Reputation: 5210

You could evaluate it with Success and Failure and create the Option that way:

rdd.flatMap{x =>
  val resTry = Try(x(2))
  resTry match {
    case Success(res)  => Some(res)
    case Failure(e) => logger.error("Error: " + Throwables.getStackTraceAsString(e))
      None
    }
}

Upvotes: 1

Related Questions