Applying filter on an RDD of Vectors/Array[Double]

Question

Suppose I have an RDD of Array[Double], with n columns. I want to apply a filter on the last column (say, the value > some constant).

• What is the syntax for this in Scala/Apache Spark?
• Is it possible to do the same if I had an RDD of Vectors?
• If I can choose one over the other (i.e., RDD of Array[Double] and RDD of Vectors), which one should I choose in order to have more efficient/faster code?

Answer 1

Something like that

val rdd: RDD[Array[Double]] = ...
val filtered: RDD[Array[Double]] = rdd.filter(arr => arr.last() > some_value)

I don't think that it really matter what to choose Array or Vector. Overall overhead of Spark is much-much higher than performance/memory benefits from Arrays vs Vectors