CODEWITHSUNDEEP
scalaapache-spark
anthonya
anthonya

Reputation: 565

Map within filter in Spark

How can I filter within mapping ?

Example : 

test1 = sc.parallelize(Array(('a', (1,Some(4)), ('b', (2, Some(5)), \
('c', (3,Some(6)), ('d',(0,None))))

What I want :

Array(('a', (1,Some(4)), ('b', (2, Some(5)), \ ('c', (3,Some(6)), \ 
('d',(613,None))))

What I tried (I've change the 0 by 613) : 

test 2 = test1.filter(value => value._2._1 == 0).mapValues(value => 
(613, value._2))

But it returns only : 

Array('d',(613,None))

Upvotes: 1

Views: 67

Answers (2)

Terry Dactyl
Terry Dactyl

Reputation: 1868

test1.map{
  case (a, (0, b)) => (a, (613, b))
  case other => other
}

Upvotes: 0

akuiper
akuiper

Reputation: 215117

Use map with pattern matching:

test1.map { 
    case (x, (0, y)) => (x, (613, y)) 
    case z => z 
}.collect
// res2: Array[(Char, (Int, Option[Int]))] = Array((a,(1,Some(4))), (b,(2,Some(5))), (c,(3,Some(6))), (d,(613,None)))

Upvotes: 3

Related Questions