CODEWITHSUNDEEP
apache-spark
Brian
Brian

Reputation: 907

How can I parallelize multiple Datasets in Spark?

I have a Spark 2.1 job where I maintain multiple Dataset objects/RDD's that represent different queries over our underlying Hive/HDFS datastore. I've noticed that if I simply iterate over the List of Datasets, they execute one at a time. Each individual query operates in parallel, but I feel that we are not maximizing our resources by not running the different datasets in parallel as well.

There doesn't seem to be a lot out there regarding doing this, as most questions appear to be around parallelizing a single RDD or Dataset, not parallelizing multiple within the same job.

Is this inadvisable for some reason? Can I just use a executor service, thread pool, or futures to do this?

Thanks!

Upvotes: 3

Views: 2095

Answers (1)

Raphael Roth
Raphael Roth

Reputation: 27373

Yes you can use multithreading in the driver code, but normally this does not increase performance, unless your queries operate on very skewed data and/or cannot be parallelized well enough to fully utilize the resources.

You can do something like that:

val datasets : Seq[Dataset[_]] = ???

datasets
  .par // transform to parallel Seq
  .foreach(ds => ds.write.saveAsTable(...)

Upvotes: 4

Related Questions