CODEWITHSUNDEEP
scalalistcase-class
James
James

Reputation: 47

Scala practices: lists and case classes

I've just started using Scala/Spark and having come from a Java background and I'm still trying to wrap my head around the concept of immutability and other best practices of Scala.

This is a very small segment of code from a larger program:
intersections is RDD(Key, (String, String))
obs is (Key, (String, String))
Data is just a case class I've defined above.

val intersections = map1 join map2
var listOfDatas = List[Data]()
intersections take NumOutputs foreach (obs => {
  listOfDatas ::= ParseInformation(obs._1.key, obs._2._1, obs._2._2)
})

listOfDatas foreach println

This code works and does what I need it to do, but I was wondering if there was a better way of making this happen. I'm using a variable list and rewriting it with a new list every single time I iterate, and I'm sure there has to be a better way to create an immutable list that's populated with the results of the ParseInformation method call. Also, I remember reading somewhere that instead of accessing the tuple values directly, the way I have done, you should use case classes within functions (as partial functions I think?) to improve readability.

Thanks in advance for any input!

Upvotes: 0

Views: 127

Answers (2)

Justin Pihony
Justin Pihony

Reputation: 67115

This might work locally, but only because you are takeing locally. It will not work once distributed as the listOfDatas is passed to each worker as a copy. The better way of doing this IMO is:

val processedData = intersections map{case (key, (item1, item2)) => {
  ParseInfo(key, item1, item2)
}}

processedData foreach println

A note for a new to functional dev: If all you are trying to do is transform data in an iterable (List), forget foreach. Use map instead, which runs your transformation on each item and spits out a new iterable of the results.

Upvotes: 4

earldouglas
earldouglas

Reputation: 13473

What's the type of intersections? It looks like you can replace foreach with map:

val listOfDatas: List[Data] =
  intersections take NumOutputs map (obs => {
    ParseInformation(obs._1.key, obs._2._1, obs._2._2)
  })

Upvotes: 1

Related Questions