CODEWITHSUNDEEP
multithreadingscalaparallel-processing
pythonic
pythonic

Reputation: 21615

How can I iterate a map in parallel in scala?

I have a code like the following.

var arr = new Array[...](map.size)
var i = 0
for((k,value) <- map)
{
    arr(i) = (k, someFunc(value))
    i += 1
}

I want this loop to execute in parallel. For example, I want it to run in 8 separate threads in parallel. How can I achieve that in Scala?

Upvotes: 2

Views: 2287

Answers (2)

Tzach Zohar
Tzach Zohar

Reputation: 37832

You can convert the map into a parallel collection, and control the number of threads by overriding the default "TaskSupport" with a pool of size 8 (or any size you want): 

import scala.collection.parallel.ForkJoinTaskSupport
import scala.collection.parallel.immutable.ParMap

val parMap: ParMap[Int, Int] = map.par
parMap.tasksupport = new ForkJoinTaskSupport(new scala.concurrent.forkjoin.ForkJoinPool(8))

parMap.foreach { case (k, value) =>
  arr(i) = (k, someFunc(value))
  i += 1
}

Do note that you can make this code more "idiomatic" by removing all the mutable values:

val arr = parMap.map { case (k, value) => (k, someFunc(value)) }.toArray
val i = arr.length

EDIT: or, an even shorter version:

val arr = parMap.mapValues(someFunc).toArray
val i = arr.length

Upvotes: 6

John K
John K

Reputation: 1285

What's that i+=1 for?

Otherwise, you are looking for (using Ints for a simple example):

scala> val m = Map(1 -> 2, 2 -> 3)
m: scala.collection.immutable.Map[Int,Int] = Map(1 -> 2, 2 -> 3)

scala> val mp = m.toParArray map {case(a,b) => (a,b+1)} 
mp: scala.collection.parallel.mutable.ParArray[(Int, Int)] = ParArray((1,3), (2,4))

scala> mp.toArray
res17: Array[(Int, Int)] = Array((1,3), (2,4))

You can just replace b+1 by someFunc(b). someFunc will be executed in parallel because we use a parallel collection (m.toParArray), and at the end we use .toArray to convert our parArray to a sequential one (not sure you need that).

Upvotes: 1

Related Questions