Scala repeated sorting in the result

Question

I have a RDD containing "Customer", "Amt spent". I am trying to perform a simple sorting to order by "Amt Spent". When i view the results, i see multiple sorting happening.

• My code:

  val sc = new SparkContext("local[*]", "TotalSpentByCustomerSorted")   
  
  val input = sc.textFile("filename.csv")
  
  val input2 = input.map(line=>line.split(",")).map(
    line=>(line(0).toInt,line(2).toFloat)
  ) // To fetch only customer#, Amt spent
  
  val input3 = input2.reduceByKey((x,y)=>(x+y)).sortBy(x=>x._2)
  
  input3.foreach(println)
  

• Result:

  (28,5000.7104)
  (22,5019.449)
  (29,5032.5303)
  (17,5032.6797)
  (60,5040.7095)
  (45,3309.3804)
  (79,3790.5698)
  (96,3924.23)
  (23,4042.65)
  (99,4172.29)
  

As you can see from the result, first 5 entries are sorted by Amt spent, and then again another sorting happens. What could be the issue here?

Answer 1

The problem is not that the sorted did nit worked, but that you called a println() inside a foreach on a RDD - that operation is performed on parallel on all partitions.
And, on a real cluster (not in a local environment for development) you would not even see the printed lines, because those would happen on the executors' JVMs.

The RDD is sorted, but if you don't want to believe me (that would be smart), then you may perform a collect before the foreach, that would fetch all results to the driver first and then print them (it will fetch them in the order they are).