CODEWITHSUNDEEP
apache-spark
Yurogini
Yurogini

Reputation: 49

When Spark finds an action does runs every line of code until the action or only the transformations that are relevant to the action?

Let 

x: RDD[Int] = {1,2,3,4}

y: RDD[Int] = {4,5,6,7}

line 1: 

x = x.map(x => x+1)

line 2: 

y = y.map(y => y+1)

line 3: 

x.collect().foreach(println)

will spark compute all the lines or only lines 1 and 3.

Upvotes: 0

Views: 66

Answers (1)

Richard Nemeth
Richard Nemeth

Reputation: 1874

In this case it will only execute lines 1 and 3. The reason why it is not executing line 2, is because you are not calling any action on RDD y, you are only building up the DAG on y.

Upvotes: 2

Related Questions