CODEWITHSUNDEEP
scalaapache-spark
Ged
Ged

Reputation: 18013

SPARK SCALA Stream ? in output

The following:

val partials = rdd.mapPartitionsWithIndex((i, iter) => {
val (keys, values) = iter.toSeq.unzip  
val sums  = values.scanLeft(0)(_ + _) 
Iterator((keys.zip(sums.tail), sums.last))   
})

partials.collect

results in:

res12: Array[(Seq[(String, Int)], Int)] = Array((Stream((c01,1), ?),10), (Stream((c05,5), ?),18), (Stream((c08,8), ?),27))

My question is: what does the "?" represent?

Upvotes: 0

Views: 74

Answers (1)

Alper t. Turker
Alper t. Turker

Reputation: 35229

Stream lazily evaluates its content (can be infinite) and memoizes visited part. Elements before ? represents already seen items, ? represents the part of the Stream which hasn't been evaluated yet.

scala> val s = Stream.continually(1)  // Infinite stream of ones
s: scala.collection.immutable.Stream[Int] = Stream(1, ?)

scala> s  // Only the head of the stream is known
res3: scala.collection.immutable.Stream[Int] = Stream(1, ?)

scala> s.take(3).foreach(_ => ())  // Evaluate the first three elements

scala> s  // Now you the first three elements are known
res5: scala.collection.immutable.Stream[Int] = Stream(1, 1, 1, ?)

Upvotes: 1

Related Questions