CODEWITHSUNDEEP
performancescala
alifirat
alifirat

Reputation: 2937

Performance issue when extracting a subset from an immutable Map

I wrote a function for my own needs taking a subset from an existing Map[String, List[Double]]. Here is the code : 

object Myobj {
  val globalPointsByChannel = 5120
  val numberOfWindow = 125
  val numOfPoints2Remove = 41
  val recordFrequency = 1024
  val windowDuration = 3000
  val localWindowSize = 3072

  def subwindow
    (
      start : Long,
      m : Map[String, List[Double]],
      nxtWindow : Map[String, List[Double]],
      left : Int,
      right : Int
    ) : List[String] = {
    for {
      windowIndex <- (0 until numberOfWindow)
    } yield {
      // in microseconds
      val startime = (start * 1000) + (math round ((windowIndex * (numOfPoints2Remove / recordFrequency) * 10e6))).toLong
      // in microseconds
      val endtime = startime + (windowDuration * 10e3).toLong
      val data : Map[String, List[Double]] = {
        for ((key, values) <- m) yield {
          // the number of points to remove at the left side from the window
          val from = windowIndex * numOfPoints2Remove
          // the number of points to remove at the right side from the window
          val until = localWindowSize + from
          val t0 = System.currentTimeMillis
          val channelData : List[Double] = values.drop(from).take(globalPointsByChannel - until - 1)
          val t1 = System.currentTimeMillis
          channelData match {
            case cd if cd.size >= localWindowSize =>
              (key -> channelData)
            case _ =>
              val numOfMissingPts = localWindowSize - channelData.size
              val missingPoints  = nxtWindow(key) take numOfMissingPts
              val completePoints : List[Double] = List(channelData,missingPoints).flatten
              (key -> completePoints)
          }
        }
      }
      val strData = data.values map (values => "[" + (values mkString(",")) + "]") mkString(",")
      strData 
    }
  }.toList
}

I think there is two parts in my code taking time. The first part : 

val channelData : List[Double] = values.drop(from).take(globalPointsByChannel - until - 1)

I'm doing this for every values in the Map for every window

The second part :

val numOfMissingPts = localWindowSize - channelData.size
          val missingPoints  = nxtWindow(key) take numOfMissingPts
          val completePoints : List[Double] = List(channelData,missingPoints).flatten
          (key -> completePoints)

This case is matched less often than the first case but again, it takes time.

For example, when I execute this code 10 times, it takes (talking in average) 9 000 msecs. Can I have some advice on how to change this two pieces of codes to make more faster than now ?

Upvotes: 1

Views: 102

Answers (1)

Andreas Neumann
Andreas Neumann

Reputation: 10894

Depending on you hardware using a ParallelCollection may work for you: http://docs.scala-lang.org/overviews/parallel-collections/overview.html

The nice thing is that you can quite easily add this to you code. Just call the .par-method on Map.

Map(1 -> "a", 2 -> "b").par
res0: scala.collection.parallel.immutable.ParMap[Int,String] = ParMap(1 -> a, 2 -> b)

Upvotes: 1

Related Questions